Infant monitoring system with observation-based system control and feedback loops

ABSTRACT

A method of optimizing sleep of a subject using smart-home devices may include operating a smart-home system that is configured to operate in a normal mode and a sleep mode. The method may also include determining that the smart-home system should transition into the sleep mode. The smart-home devices may use a set of default parameters when operating in the sleep mode. The method may additionally include monitoring, while in the sleep mode, a sleep cycle of the subject using the smart-home devices. The method may further include detecting behavior of the subject that indicates that the sleep cycle of the subject is being interrupted or about to be interrupted, determining an environmental control that corresponds with the behavior of the subject, and adjusting the environmental control using the smart-home devices to prevent or stop the sleep cycle of the subject from being interrupted.

CROSS REFERENCE TO RELATED APPLICATIONS

This patent application is related to U.S. patent application Ser. No.15/859,640, entitled “ENHANCED VISUALIZATION OF BREATHING OR HEARTBEATOF AN INFANT OR OTHER MONITORED SUBJECT” filed concurrently with thepresent application on Dec. 31, 2017 (Attorney Docket No.094021-1064573), which is hereby incorporated by reference in itsentirety for all purposes.

This patent application is also related to U.S. patent application Ser.No. 15/859,650, entitled “INFANT MONITORING SYSTEM WITH VIDEO-BASEDTEMPERATURE BASELINING AND ELEVATED TEMPERATURE DETECTION” filedconcurrently with the present application on Dec. 31, 2017 (AttorneyDocket No. 094021-1064636), which is hereby incorporated by reference inits entirety for all purposes.

TECHNICAL FIELD

This patent specification relates generally to a smart-home environmentfor monitoring subject. More particularly, this patent specificationdescribes automatic control of smart-home devices, such as video cameraassemblies, keypads, security system sensors, thermostats, hazarddetectors, doorbells, and/or the like, to create an optimal sleepenvironment for a monitored subject.

BACKGROUND

Smart-home devices are rapidly becoming part of the modern homeexperience. These devices may include thermostats, keypads, touchscreens, and/or other control devices for controlling environmentalsystems, such as HVAC systems or lighting systems. The smart-homeenvironment may also include smart appliances, such as washing machines,dishwashers, refrigerators, garbage cans, and so forth, that interfacewith control and/or monitoring devices to increase the level offunctionality and control provided to an occupant. Security systems,including cameras, keypads, sensors, motion detectors, glass-breaksensors, microphones, and so forth, may also be installed as part of thesmart-home architecture.

Other smart-the home devices may include doorbells, monitoring systems,hazard detectors, smart lightbulbs, and virtually any other electronicdevice that can be controlled via a wired/wireless network.

Many modern smart-home environments may include video cameras. Thesevideo cameras may be used for security systems, monitoring systems,hazard detection systems, and so forth. In general, video camerasprovide a live video feed that can be played at a local console or on acomputing system of the user, allowing them to remotely monitor aportion of the smart-home environment or its surroundings.

BRIEF SUMMARY

In some embodiments, a method of monitoring and optimizing the sleep ofa subject using a plurality of smart-home devices may include operatinga smart-home system comprising the plurality of smart-home devices. Thesmart-home system may be configured to operate in a plurality of modesincluding a normal operating mode and a sleep mode. The method may alsoinclude determining that the smart-home system should transition intothe sleep mode from the normal operating mode. The plurality ofsmart-home devices may use a set of default parameters when operating inthe sleep mode. The method may additionally include monitoring, while inthe sleep mode, a sleep cycle of the subject using the plurality ofsmart-home devices. The method may further include detecting, by theplurality of smart-home devices, behavior of the subject that indicatesthat the sleep cycle of the subject is being interrupted or about to beinterrupted, and determining an environmental control that correspondswith the behavior of the subject. The method may also include adjustingthe environmental control using the plurality of smart-home devices. Theadjusting may be configured to prevent or stop the sleep cycle of thesubject from being interrupted.

In some embodiments, a smart-home system for monitoring and optimizingthe sleep of a subject using a plurality of smart-home devices mayinclude a plurality of smart-home devices that are configured to operatein a plurality of modes including a normal operating mode and a sleepmode. The system may also include one or more processors and one or morememory devices comprising instructions that, when executed by the one ormore processors, cause the one or more processors to perform operationsthat may include determining that the smart-home system shouldtransition into the sleep mode from the normal operating mode. Theplurality of smart-home devices may use a set of default parameters whenoperating in the sleep mode. The operations may additionally includemonitoring, while in the sleep mode, a sleep cycle of the subject usingthe plurality of smart-home devices. The operations may further includedetecting, by the plurality of smart-home devices, behavior of thesubject that indicates that the sleep cycle of the subject is beinginterrupted or about to be interrupted, and determining an environmentalcontrol that corresponds with the behavior of the subject. Theoperations may also include adjusting the environmental control usingthe plurality of smart-home devices. The adjusting may be configured toprevent or stop the sleep cycle of the subject from being interrupted.

A further understanding of the nature and advantages of the presentinvention may be realized by reference to the remaining portions of thespecification and the drawings. Also note that other embodiments may bedescribed in the following disclosure and claims.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is an example of a smart-home environment within which one ormore of the devices, methods, systems, services, and/or computer programproducts described further herein will be applicable, according to someembodiments.

FIG. 2A illustrates a simplified block diagram of a representativenetwork architecture that includes a smart-home network in accordance,according to some embodiments.

FIG. 2B illustrates a simplified operating environment in which a serversystem interacts with client devices and smart devices, according tosome embodiments.

FIG. 3 illustrates a block diagram of a representative smart device inaccordance with some implementations.

FIG. 4A illustrates a view of a representative camera assembly inaccordance with some implementations.

FIG. 4B illustrates a view of a representative camera assembly inaccordance with some implementations.

FIG. 5A is an expanded component view of a representative cameraassembly in accordance with some implementations.

FIG. 5B is an expanded component view of a representative cameraassembly in accordance with some implementations.

FIG. 6 illustrates an infant sleeping in a sleep environment and beingmonitored by a camera, according to some embodiments.

FIG. 7 illustrates a flowchart of a method for determining when thesmart-home devices in the home should enter into a sleep mode, accordingto some embodiments.

FIG. 8 illustrates a view of the infant that may be received by thecamera, according to some embodiments.

FIG. 9 illustrates a camera view and of the infant with a bounding boxthat reduces the processing power, memory, and/or bandwidth required bythe system, according to some embodiments.

FIG. 10 illustrates a system for providing default parameters forsmart-home devices, according to some embodiments.

FIG. 11 illustrates a flowchart of a method for monitoring the sleep ofthe subject and adjusting environmental conditions to optimize the sleepenvironment, according to some embodiments.

FIG. 12 illustrates a flowchart of a method for automatically soothing asubject whose sleep has been interrupted, according to some embodiments.

FIG. 13 illustrates an image similar to that of FIG. 8 captured by thethermal imager function of the camera, according to some embodiments.

FIG. 14 illustrates a thermal image that can diagnose an infant that isto cold, according to some embodiments.

FIG. 15 illustrates a thermal image that can be used to diagnose aninfant who is too warm, according to some embodiments.

FIG. 16 illustrates how some of the sleep aids can be activated in thesleep environment, according to some embodiments.

FIG. 17 illustrates how light can be used as a sleep aid to auto-soothethe infant, according to some embodiments.

FIG. 18 illustrates a system diagram for processing and transmittingimages and information between the smart-home devices in the sleepenvironment and a user's mobile device, according to some embodiments.

FIG. 19 illustrates a representation of the thermal video feed as it isdisplayed on a mobile device.

FIG. 20 illustrates a representation of the live video feed displayed ona mobile device, according to some embodiments.

FIG. 21 illustrates an alternative visual representation of an alert ona mobile device 166-1, according to some embodiments.

FIG. 22 illustrates a closed-loop local feedback system for updatingdefault sleep-mode parameters, according to some embodiments.

FIG. 23 illustrates a crowd-sourced feedback system for updating defaultsleep parameters for populations of similar subjects, according to someembodiments.

DETAILED DESCRIPTION

In the following detailed description, for purposes of explanation,numerous specific details are set forth to provide a thoroughunderstanding of the various embodiments of the present invention. Thoseof ordinary skill in the art will realize that these various embodimentsof the present invention are illustrative only and are not intended tobe limiting in any way. Other embodiments of the present invention willreadily suggest themselves to such skilled persons having the benefit ofthis disclosure. It will be apparent to one skilled in the art that thepresent invention may be practiced without some or all of these specificdetails. In other instances, well known details have not been describedin detail in order not to unnecessarily obscure the present invention.

In addition, for clarity purposes, not all of the routine features ofthe embodiments described herein are shown or described. One of ordinaryskill in the art would readily appreciate that in the development of anysuch actual embodiment, numerous embodiment-specific decisions may berequired to achieve specific design objectives. These design objectiveswill vary from one embodiment to another and from one developer toanother. Moreover, it will be appreciated that such a development effortmight be complex and time-consuming but would nevertheless be a routineengineering undertaking for those of ordinary skill in the art havingthe benefit of this disclosure.

FIG. 1 illustrates an example smart-home environment 100, according tosome embodiments. The smart-home environment 100 includes a structure150 (e.g., a house, office building, garage, or mobile home) withvarious integrated devices. It will be appreciated that devices may alsobe integrated into a smart-home environment 100 that does not include anentire structure 150, such as an apartment, condominium, or officespace. Further, the smart-home environment 100 may control and/or becoupled to devices outside of the actual structure 150. Indeed, severaldevices in the smart-home environment 100 need not be physically withinthe structure 150. For example, a device controlling a pool heater 114or irrigation system 116 may be located outside of the structure 150.

The term “smart-home environment” may refer to smart environments forhomes such as a single-family house, but the scope of the presentteachings is not so limited. The present teachings are also applicable,without limitation, to duplexes, townhomes, multi-unit apartmentbuildings, hotels, retail stores, office buildings, industrialbuildings, and more generally any living space or work space. Similarly,while the terms user, customer, installer, homeowner, occupant, guest,tenant, landlord, repair person, etc., may be used to refer to a personor persons acting in the context of some particular situations describedherein, these references do not limit the scope of the present teachingswith respect to the person or persons who are performing such actions.Thus, for example, the terms user, customer, purchaser, installer,subscriber, and homeowner may often refer to the same person in the caseof a single-family residential dwelling, because the head of thehousehold is often the person who makes the purchasing decision, buysthe unit, and installs and configures the unit, as well as being one ofthe users of the unit. However, in other scenarios, such as alandlord-tenant environment, the customer may be the landlord withrespect to purchasing the unit, the installer may be a local apartmentsupervisor, a first user may be the tenant, and a second user may againbe the landlord with respect to remote control functionality. While theidentity of the person performing the action may be germane to aparticular advantage provided by one or more of the implementations,such an identity should not be construed in the descriptions that followas necessarily limiting the scope of the present teachings to thoseparticular individuals having those particular identities.

The depicted structure 150 includes a plurality of rooms 152, separatedat least partly from each other via walls 154. The walls 154 may includeinterior walls or exterior walls. Each room may further include a floor156 and a ceiling 158. Devices may be mounted on, integrated with and/orsupported by a wall 154, floor 156, or ceiling 158.

In some implementations, the integrated devices of the smart-homeenvironment 100 include intelligent, multi-sensing, network-connecteddevices that integrate seamlessly with each other in a smart-homenetwork and/or with a central server or a cloud-computing system toprovide a variety of useful smart-home functions. The smart-homeenvironment 100 may include one or more intelligent, multi-sensing,network-connected thermostats 102 (hereinafter referred to as “smartthermostats 102”), one or more intelligent, network-connected,multi-sensing hazard detection units 104 (hereinafter referred to as“smart hazard detectors 104”), one or more intelligent, multi-sensing,network-connected entryway interface devices 106 and 120 (hereinafterreferred to as “smart doorbells 106” and “smart door locks 120”), andone or more intelligent, multi-sensing, network-connected alarm systems122 (hereinafter referred to as “smart alarm systems 122”). Although notdepicted explicitly in FIG. 1, the smart-home environment 100 may alsoinclude other monitoring systems, such as baby monitoring systems,elderly monitoring systems, handicapped monitoring systems, and soforth.

In some implementations, the one or more smart thermostats 102 detectambient climate characteristics (e.g., temperature and/or humidity) andcontrol a HVAC system 103 accordingly. For example, a respective smartthermostat 102 includes an ambient temperature sensor.

The one or more smart hazard detectors 104 may include thermal radiationsensors directed at respective heat sources (e.g., a stove, oven, otherappliances, a fireplace, etc.). For example, a smart hazard detector 104in a kitchen 153 may include a thermal radiation sensor directed at astove/oven 112. A thermal radiation sensor may determine the temperatureof the respective heat source (or a portion thereof) at which it isdirected and may provide corresponding blackbody radiation data asoutput.

The smart doorbell 106 and/or the smart door lock 120 may detect aperson's approach to or departure from a location (e.g., an outer door),control doorbell/door locking functionality (e.g., receive user inputsfrom a portable electronic device 166-1 to actuate bolt of the smartdoor lock 120), announce a person's approach or departure via audio orvisual devices, and/or control settings on a security system (e.g., toactivate or deactivate the security system when occupants go and come).In some implementations, the smart doorbell 106 may include some or allof the components and features of the camera 118. In someimplementations, the smart doorbell 106 includes a camera 118.

The smart alarm system 122 may detect the presence of an individualwithin close proximity (e.g., using built-in IR sensors), sound an alarm(e.g., through a built-in speaker, or by sending commands to one or moreexternal speakers), and send notifications to entities or userswithin/outside of the smart-home network 100. In some implementations,the smart alarm system 122 also includes one or more input devices orsensors (e.g., keypad, biometric scanner, NFC transceiver, microphone)for verifying the identity of a user, and one or more output devices(e.g., display, speaker) for providing notifications. In someimplementations, the smart alarm system 122 may also be set to an“armed” mode, such that detection of a trigger condition or event causesthe alarm to be sounded unless a disarming action is performed.

In some implementations, the smart-home environment 100 may include oneor more intelligent, multi-sensing, network-connected wall switches 108(hereinafter referred to as “smart wall switches 108”), along with oneor more intelligent, multi-sensing, network-connected wall pluginterfaces 110 (hereinafter referred to as “smart wall plugs 110”). Thesmart wall switches 108 may detect ambient lighting conditions, detectroom-occupancy states, and control a power and/or dim state of one ormore lights. In some instances, smart wall switches 108 may also controla power state or speed of a fan, such as a ceiling fan. The smart wallplugs 110 may detect occupancy of a room or enclosure and control supplyof power to one or more wall plugs (e.g., such that power is notsupplied to the plug if nobody is at home).

In some implementations, the smart-home environment 100 of FIG. 1 mayinclude a plurality of intelligent, multi-sensing, network-connectedappliances 112 (hereinafter referred to as “smart appliances 112”), suchas refrigerators, stoves, ovens, televisions, washers, dryers, lights,stereos, intercom systems, garage-door openers, floor fans, ceilingfans, wall air conditioners, pool heaters, irrigation systems, securitysystems, space heaters, window AC units, motorized duct vents, and soforth. In some implementations, when plugged in, an appliance mayannounce itself to the smart home network, such as by indicating whattype of appliance it is, and it may automatically integrate with thecontrols of the smart home. Such communication by the appliance to thesmart home may be facilitated by either a wired or wirelesscommunication protocol. The smart home may also include a variety ofnon-communicating legacy appliances 140, such as older-modelconventional washers/dryers, refrigerators, and/or the like, which maybe controlled by smart wall plugs 110. The smart-home environment 100may further include a variety of partially communicating legacyappliances 142, such as infrared (“IR”) controlled wall air conditionersor other IR-controlled devices, which may be controlled by IR signalsprovided by the smart hazard detectors 104, hand-held remote controls,key FOBs, or the smart wall switches 108.

In some implementations, the smart-home environment 100 may include oneor more network-connected cameras 118 that are configured to providevideo monitoring and security in the smart-home environment 100. Thecameras 118 may be used to determine the occupancy of the structure 150and/or particular rooms 152 in the structure 150, and thus may act asoccupancy sensors. For example, video captured by the cameras 118 may beprocessed to identify the presence of an occupant in the structure 150(e.g., in a particular room 152). Specific individuals may be identifiedbased, for example, on their appearance (e.g., height, face) and/ormovement (e.g., their walk/gait). Cameras 118 may additionally includeone or more sensors (e.g., IR sensors, motion detectors), input devices(e.g., microphone for capturing audio), and output devices (e.g.,speaker for outputting audio). In some implementations, the cameras 118may each be configured to operate in a day mode and in a low-light mode(e.g., a night mode). In some implementations, the cameras 118 eachinclude one or more IR illuminators for providing illumination while thecamera is operating in the low-light mode. In some implementations, thecameras 118 include one or more outdoor cameras. In someimplementations, the outdoor cameras include additional features and/orcomponents such as weatherproofing and/or solar ray compensation.

The smart-home environment 100 may additionally or alternatively includeone or more other occupancy sensors (e.g., the smart doorbell 106, smartdoor locks 120, touch screens, IR sensors, microphones, ambient lightsensors, motion detectors, smart nightlights 170, etc.). In someimplementations, the smart-home environment 100 may includeradio-frequency identification (RFID) readers (e.g., in each room 152 ora portion thereof) that determine occupancy based on RFID tags locatedon or embedded in occupants. For example, RFID readers may be integratedinto the smart hazard detectors 104, and RFID tags may be worn in usersclothing for integrated in hand-held devices such as a smart phone.

The smart-home environment 100 may also include communication withdevices outside of the physical home but within a proximate geographicalrange of the home. For example, the smart-home environment 100 mayinclude a pool heater monitor 114 that communicates a current pooltemperature to other devices within the smart-home environment 100and/or receives commands for controlling the pool temperature.Similarly, the smart-home environment 100 may include an irrigationmonitor 116 that communicates information regarding irrigation systemswithin the smart-home environment 100 and/or receives controlinformation for controlling such irrigation systems.

By virtue of network connectivity, one or more of the smart home devicesof FIG. 1 may further allow a user to interact with the device even ifthe user is not proximate to the device. For example, a user maycommunicate with a device using a computer (e.g., a desktop computer,laptop computer, or tablet) or other portable electronic device 166(e.g., a mobile phone, such as a smart phone). A webpage or applicationmay be configured to receive communications from the user and controlthe device based on the communications and/or to present informationabout the device's operation to the user. For example, the user may viewa current set point temperature for a device (e.g., a stove) and adjustit using a computer. The user may be in the structure during this remotecommunication or outside the structure.

As discussed above, users may control smart devices in the smart-homeenvironment 100 using a network-connected computer or portableelectronic device 166. In some examples, some or all of the occupants(e.g., individuals who live in the home) may register their device 166with the smart-home environment 100. Such registration may be made at acentral server to authenticate the occupant and/or the device as beingassociated with the home and to give permission to the occupant to usethe device to control the smart devices in the home. An occupant may usetheir registered device 166 to remotely control the smart devices of thehome, such as when the occupant is at work or on vacation. The occupantmay also use their registered device to control the smart devices whenthe occupant is actually located inside the home, such as when theoccupant is sitting on a couch inside the home. It should be appreciatedthat instead of or in addition to registering devices 166, thesmart-home environment 100 may make inferences about (1) whichindividuals live in the home and are therefore occupants, and (2) whichdevices 166 are associated with those individuals. As such, thesmart-home environment may “learn” who is an occupant and permit thedevices 166 associated with those individuals to control the smartdevices of the home.

In some implementations, in addition to containing processing andsensing capabilities, devices 102, 104, 106, 108, 110, 112, 114, 116,118, 120, and/or 122 (collectively referred to as “the smart devices” or“the smart-home devices”) are capable of data communications andinformation sharing with other smart devices, a central server orcloud-computing system, and/or other devices that are network-connected.Data communications may be carried out using any of a variety of customor standard wireless protocols (e.g., IEEE 802.15.4, Wi-Fi, ZigBee,6LoWPAN, Thread, Z-Wave, Bluetooth Smart, ISA100.5A, WirelessHART, MiWi,etc.) and/or any of a variety of custom or standard wired protocols(e.g., Ethernet, HomePlug, etc.), or any other suitable communicationprotocol, including communication protocols not yet developed as of thefiling date of this document.

In some implementations, the smart devices may serve as wireless orwired repeaters. In some implementations, a first one of the smartdevices communicates with a second one of the smart devices via awireless router. The smart devices may further communicate with eachother via a connection (e.g., network interface 160) to a network, suchas the Internet 162. Through the Internet 162, the smart devices maycommunicate with a server system 164 (also called a central serversystem and/or a cloud-computing system herein). The server system 164may be associated with a manufacturer, support entity, or serviceprovider associated with the smart device(s). In some implementations, auser is able to contact customer support using a smart device itselfrather than needing to use other communication means, such as atelephone or Internet-connected computer. In some implementations,software updates are automatically sent from the server system 164 tosmart devices (e.g., when available, when purchased, or at routineintervals).

In some implementations, the network interface 160 includes aconventional network device (e.g., a router), and the smart-homeenvironment 100 of FIG. 1 includes a hub device 180 that iscommunicatively coupled to the network(s) 162 directly or via thenetwork interface 160. The hub device 180 may be further communicativelycoupled to one or more of the above intelligent, multi-sensing,network-connected devices (e.g., smart devices of the smart-homeenvironment 100). Each of these smart devices optionally communicateswith the hub device 180 using one or more radio communication networksavailable at least in the smart-home environment 100 (e.g., ZigBee,Z-Wave, Insteon, Bluetooth, Wi-Fi and other radio communicationnetworks). In some implementations, the hub device 180 and devicescoupled with/to the hub device can be controlled and/or interacted withvia an application running on a smart phone, household controller,laptop, tablet computer, game console or similar electronic device. Insome implementations, a user of such controller application can viewstatus of the hub device or coupled smart devices, configure the hubdevice to interoperate with smart devices newly introduced to the homenetwork, commission new smart devices, and adjust or view settings ofconnected smart devices, etc. In some implementations the hub deviceextends the capabilities of low-capability smart devices to match thecapabilities of the highly capable smart devices of the same type,integrates functionality of multiple different device types—even acrossdifferent communication protocols, and is configured to streamlineadding of new devices and commissioning of the hub device. In someimplementations, hub device 180 further comprises a local storage devicefor storing data related to, or output by, smart devices of smart-homeenvironment 100. In some implementations, the data includes one or moreof: video data output by a camera device, metadata output by a smartdevice, settings information for a smart device, usage logs for a smartdevice, and the like.

In some implementations, smart-home environment 100 includes a localstorage device 190 for storing data related to, or output by, smartdevices of smart-home environment 100. In some implementations, the dataincludes one or more of: video data output by a camera device (e.g.,camera 118), metadata output by a smart device, settings information fora smart device, usage logs for a smart device, and the like. In someimplementations, local storage device 190 is communicatively coupled toone or more smart devices via a smart home network. In someimplementations, local storage device 190 is selectively coupled to oneor more smart devices via a wired and/or wireless communication network.In some implementations, local storage device 190 is used to store videodata when external network conditions are poor. For example, localstorage device 190 is used when an encoding bitrate of camera 118exceeds the available bandwidth of the external network (e.g.,network(s) 162). In some implementations, local storage device 190temporarily stores video data from one or more cameras (e.g., camera118) prior to transferring the video data to a server system (e.g.,server system 164).

In some implementations, the smart-home environment 100 includes servicerobots 168 that are configured to carry out, in an autonomous manner,any of a variety of household tasks.

FIG. 2A illustrates a simplified block diagram of a representativenetwork architecture 200 that includes a smart home network 202 inaccordance with some implementations. In some implementations, the smartdevices 204 in the smart-home environment 100 (e.g., devices 102, 104,106, 108, 110, 112, 114, 116, 118, 120, and/or 122) combine with the hubdevice 180 to create a mesh network in smart home network 202. In someimplementations, one or more smart devices 204 in the smart home network202 operate as a smart home controller. Additionally and/oralternatively, hub device 180 operates as the smart home controller. Insome implementations, a smart home controller has more computing powerthan other smart devices. In some implementations, a smart homecontroller processes inputs (e.g., from smart devices 204, electronicdevice 166, and/or server system 164) and sends commands (e.g., to smartdevices 204 in the smart home network 202) to control operation of thesmart-home environment 100. In some implementations, some of the smartdevices 204 in the smart home network 202 (e.g., in the mesh network)are “spokesman” nodes (e.g., 204-1) and others are “low-powered” nodes(e.g., 204-9). Some of the smart devices in the smart-home environment100 are battery powered, while others have a regular and reliable powersource, such as by connecting to wiring (e.g., to 120V line voltagewires) behind the walls 154 of the smart-home environment. The smartdevices that have a regular and reliable power source are referred to as“spokesman” nodes. These nodes are typically equipped with thecapability of using a wireless protocol to facilitate bidirectionalcommunication with a variety of other devices in the smart-homeenvironment 100, as well as with the server system 164. In someimplementations, one or more “spokesman” nodes operate as a smart homecontroller. On the other hand, the devices that are battery powered arethe “low-power” nodes. These nodes tend to be smaller than spokesmannodes and typically only communicate using wireless protocols thatrequire very little power, such as Zigbee, ZWave, 6LoWPAN, Thread,Bluetooth, etc.

In some implementations, some low-power nodes may be incapable ofbidirectional communication. These low-power nodes may send messages,but they are unable to “listen.” Thus, other devices in the smart-homeenvironment 100, such as the spokesman nodes, need not send informationto these low-power nodes. In some implementations, some low-power nodesare capable of only a limited bidirectional communication. For example,other devices are able to communicate with the low-power nodes onlyduring a certain time period.

In some implementations, the smart devices may serve as low-power andspokesman nodes to create a mesh network in the smart-home environment100. In some implementations, individual low-power nodes in thesmart-home environment may regularly send out messages regarding whatthey are sensing, and the other low-powered nodes in the smart-homeenvironment—in addition to sending out their own messages—may forwardthese messages, thereby causing the messages to travel from node to node(i.e., device to device) throughout the smart home network 202. In someimplementations, the spokesman nodes in the smart home network 202,which are able to communicate using a relatively high-powercommunication protocol, such as IEEE 802.11, are able to switch to arelatively low-power communication protocol, such as IEEE 802.15.4, toreceive these messages, translate the messages to other communicationprotocols, and send the translated messages to other spokesman nodesand/or the server system 164 (using, e.g., the relatively high-powercommunication protocol). Thus, the low-powered nodes using low-powercommunication protocols are able to send and/or receive messages acrossthe entire smart home network 202, as well as over the Internet 162 tothe server system 164. In some implementations, the mesh network enablesthe server system 164 to regularly receive data from most or all of thesmart devices in the home, make inferences based on the data, facilitatestate synchronization across devices within and outside of the smarthome network 202, and send commands to one or more of the smart devicesto perform tasks in the smart-home environment.

The spokesman nodes and some of the low-powered nodes are capable of“listening.” Accordingly, users, other devices, and/or the server system164 may communicate control commands to the low-powered nodes. Forexample, a user may use the electronic device 166 (e.g., a smart phone)to send commands over the Internet to the server system 164, which thenrelays the commands to one or more spokesman nodes in the smart homenetwork 202. The spokesman nodes may use a low-power protocol tocommunicate the commands to the low-power nodes throughout the smarthome network 202, as well as to other spokesman nodes that did notreceive the commands directly from the server system 164.

In some implementations, a smart nightlight 170, which is an example ofa smart device 204, is a low-power node. In addition to housing a lightsource, the smart nightlight 170 houses an occupancy sensor, such as anultrasonic or passive IR sensor, and an ambient light sensor, such as aphoto resistor or a single-pixel sensor that measures light in the room.In some implementations, the smart nightlight 170 is configured toactivate the light source when its ambient light sensor detects that theroom is dark and when its occupancy sensor detects that someone is inthe room. In other implementations, the smart nightlight 170 is simplyconfigured to activate the light source when its ambient light sensordetects that the room is dark. Further, in some implementations, thesmart nightlight 170 includes a low-power wireless communication chip(e.g., a ZigBee chip) that regularly sends out messages regarding theoccupancy of the room and the amount of light in the room, includinginstantaneous messages coincident with the occupancy sensor detectingthe presence of a person in the room. As described above, these messagesmay be sent wirelessly (e.g., using the mesh network) from node to node(i.e., smart device to smart device) within the smart home network 202as well as over the Internet 162 to the server system 164.

Other examples of low-power nodes include battery-operated versions ofthe smart hazard detectors 104. These smart hazard detectors 104 areoften located in an area without access to constant and reliable powerand may include any number and type of sensors, such as smoke/fire/heatsensors (e.g., thermal radiation sensors), carbon monoxide/dioxidesensors, occupancy/motion sensors, ambient light sensors, ambienttemperature sensors, humidity sensors, and the like. Furthermore, smarthazard detectors 104 may send messages that correspond to each of therespective sensors to the other devices and/or the server system 164,such as by using the mesh network as described above.

Examples of spokesman nodes include smart doorbells 106, smartthermostats 102, smart wall switches 108, and smart wall plugs 110.These devices are often located near and connected to a reliable powersource, and therefore may include more power-consuming components, suchas one or more communication chips capable of bidirectionalcommunication in a variety of protocols.

As explained above with reference to FIG. 1, in some implementations,the smart-home environment 100 of FIG. 1 includes a hub device 180 thatis communicatively coupled to the network(s) 162 directly or via thenetwork interface 160. The hub device 180 is further communicativelycoupled to one or more of the smart devices using a radio communicationnetwork that is available at least in the smart-home environment 100.Communication protocols used by the radio communication network include,but are not limited to, ZigBee, Z-Wave, Insteon, EuOcean, Thread, OSIAN,Bluetooth Low Energy and the like. In some implementations, the hubdevice 180 not only converts the data received from each smart device tomeet the data format requirements of the network interface 160 or thenetwork(s) 162, but also converts information received from the networkinterface 160 or the network(s) 162 to meet the data format requirementsof the respective communication protocol associated with a targetedsmart device. In some implementations, in addition to data formatconversion, the hub device 180 further processes the data received fromthe smart devices or information received from the network interface 160or the network(s) 162 preliminary. For example, the hub device 180 canintegrate inputs from multiple sensors/connected devices (includingsensors/devices of the same and/or different types), perform higherlevel processing on those inputs—e.g., to assess the overall environmentand coordinate operation among the different sensors/devices—and/orprovide instructions to the different devices based on the collection ofinputs and programmed processing. It is also noted that in someimplementations, the network interface 160 and the hub device 180 areintegrated to one network device. Functionality described herein isrepresentative of particular implementations of smart devices, controlapplication(s) running on representative electronic device(s) (such as asmart phone), hub device(s) 180, and server(s) coupled to hub device(s)via the Internet or other Wide Area Network (WAN). All or a portion ofthis functionality and associated operations can be performed by anyelements of the described system—for example, all or a portion of thefunctionality described herein as being performed by an implementationof the hub device can be performed, in different system implementations,in whole or in part on the server, one or more connected smart devicesand/or the control application, or different combinations thereof.

FIG. 2B illustrates a representative operating environment in which aserver system 164 provides data processing for monitoring andfacilitating review of events (e.g., motion, audio, security, etc.) invideo streams captured by video cameras 118. As shown in FIG. 2B, theserver system 164 receives video data from video sources 222 (includingcameras 118) located at various physical locations (e.g., inside homes,restaurants, stores, streets, parking lots, and/or the smart-homeenvironments 100 of FIG. 1). Each video source 222 may be bound to oneor more reviewer accounts, and the server system 164 provides videomonitoring data for the video source 222 to client devices 220associated with the reviewer accounts. For example, the portableelectronic device 166 is an example of the client device 220. In someimplementations, the server system 164 is a video processing server thatprovides video processing services to video sources and client devices220.

In some implementations, each of the video sources 222 includes one ormore video cameras 118 that capture video and send the captured video tothe server system 164 substantially in real-time. In someimplementations, each of the video sources 222 includes a controllerdevice (not shown) that serves as an intermediary between the one ormore cameras 118 and the server system 164. The controller devicereceives the video data from the one or more cameras 118, optionallyperforms some preliminary processing on the video data, and sends thevideo data to the server system 164 on behalf of the one or more cameras118 substantially in real-time. In some implementations, each camera hasits own on-board processing capabilities to perform some preliminaryprocessing on the captured video data before sending the processed videodata (along with metadata obtained through the preliminary processing)to the controller device and/or the server system 164.

In accordance with some implementations, each of the client devices 220includes a client-side module. The client-side module communicates witha server-side module executed on the server system 164 through the oneor more networks 162. The client-side module provides client-sidefunctionality for the event monitoring and review processing andcommunications with the server-side module. The server-side moduleprovides server-side functionality for event monitoring and reviewprocessing for any number of client-side modules each residing on arespective client device 220. The server-side module also providesserver-side functionality for video processing and camera control forany number of the video sources 222, including any number of controldevices and the cameras 118.

In some implementations, the server system 164 includes one or moreprocessors 212, a video storage database 210, an account database 214,an I/O interface to one or more client devices 216, and an I/O interfaceto one or more video sources 218. The I/O interface to one or moreclients 216 facilitates the client-facing input and output processing.The account database 214 stores a plurality of profiles for revieweraccounts registered with the video processing server, where a respectiveuser profile includes account credentials for a respective revieweraccount, and one or more video sources linked to the respective revieweraccount. The I/O interface to one or more video sources 218 facilitatescommunications with one or more video sources 222 (e.g., groups of oneor more cameras 118 and associated controller devices). The videostorage database 210 stores raw video data received from the videosources 222, as well as various types of metadata, such as motionevents, event categories, event category models, event filters, andevent masks, for use in data processing for event monitoring and reviewfor each reviewer account.

Examples of a representative client device 220 include a handheldcomputer, a wearable computing device, a personal digital assistant(PDA), a tablet computer, a laptop computer, a desktop computer, acellular telephone, a smart phone, an enhanced general packet radioservice (EGPRS) mobile phone, a media player, a navigation device, agame console, a television, a remote control, a point-of-sale (POS)terminal, a vehicle-mounted computer, an eBook reader, or a combinationof any two or more of these data processing devices or other dataprocessing devices.

Examples of the one or more networks 162 include local area networks(LAN) and wide area networks (WAN) such as the Internet. The one or morenetworks 162 are implemented using any known network protocol, includingvarious wired or wireless protocols, such as Ethernet, Universal SerialBus (USB), FIREWIRE, Long Term Evolution (LTE), Global System for MobileCommunications (GSM), Enhanced Data GSM Environment (EDGE), codedivision multiple access (CDMA), time division multiple access (TDMA),Bluetooth, Wi-Fi, voice over Internet Protocol (VoIP), Wi-MAX, or anyother suitable communication protocol.

In some implementations, the server system 164 may be implemented on oneor more standalone data processing apparatuses or a distributed networkof computers. In some implementations, the server system 164 alsoemploys various virtual devices and/or services of third party serviceproviders (e.g., third-party cloud service providers) to provide theunderlying computing resources and/or infrastructure resources of theserver system 164. In some implementations, the server system 164includes, but is not limited to, a server computer, a handheld computer,a tablet computer, a laptop computer, a desktop computer, or acombination of any two or more of these data processing devices or otherdata processing devices.

The server-client environment shown in FIG. 2B includes both aclient-side portion (e.g., the client-side module) and a server-sideportion (e.g., the server-side module). The division of functionalitybetween the client and server portions of operating environment can varyin different implementations. Similarly, the division of functionalitybetween a video source 222 and the server system 164 can vary indifferent implementations. For example, in some implementations, theclient-side module is a thin-client that provides only user-facing inputand output processing functions, and delegates all other data processingfunctionality to a backend server (e.g., the server system 164).Similarly, in some implementations, a respective one of the videosources 222 is a simple video capturing device that continuouslycaptures and streams video data to the server system 164 with limited orno local preliminary processing on the video data. Although many aspectsof the present technology are described from the perspective of theserver system 164, the corresponding actions performed by a clientdevice 220 and/or the video sources 222 would be apparent to one ofskill in the art.

Similarly, some aspects of the present technology may be described fromthe perspective of a client device or a video source, and thecorresponding actions performed by the video server would be apparent toone of skill in the art. Furthermore, some aspects of the presenttechnology may be performed by the server system 164, a client device220, and a video source 222 cooperatively.

In some implementations, a video source 222 (e.g., a camera 118)transmits one or more streams of video data to the server system 164. Insome implementations, the one or more streams may include multiplestreams, of respective resolutions and/or frame rates, of the raw videocaptured by the camera 118. In some implementations, the multiplestreams may include a “primary” stream with a certain resolution andframe rate, corresponding to the raw video captured by the camera 118,and one or more additional streams. An additional stream may be the samevideo stream as the “primary” stream but at a different resolutionand/or frame rate, or a stream that captures a portion of the “primary”stream (e.g., cropped to include a portion of the field of view orpixels of the primary stream) at the same or different resolution and/orframe rate as the “primary” stream.

In some implementations, one or more of the streams are sent from thevideo source 222 directly to a client device 220 (e.g., without beingrouted to, or processed by, the server system 164). In someimplementations, one or more of the streams is stored at the camera 118(e.g., in memory 406, FIG. 4) and/or a local storage device (e.g., adedicated recording device), such as a digital video recorder (DVR). Forexample, in accordance with some implementations, the camera 118 storesthe most recent 24 hours of video footage recorded by the camera. Insome implementations, portions of the one or more streams are stored atthe camera 118 and/or the local storage device (e.g., portionscorresponding to particular events or times of interest).

In some implementations, the server system 164 transmits one or morestreams of video data to a client device 220 to facilitate eventmonitoring by a user. In some implementations, the one or more streamsmay include multiple streams, of respective resolutions and/or framerates, of the same video feed. In some implementations, the multiplestreams may include a “primary” stream with a certain resolution andframe rate, corresponding to the video feed, and one or more additionalstreams. An additional stream may be the same video stream as the“primary” stream but at a different resolution and/or frame rate, or astream that shows a portion of the “primary” stream (e.g., cropped toinclude portion of the field of view or pixels of the primary stream) atthe same or different resolution and/or frame rate as the “primary”stream, as described in greater detail in U.S. patent application Ser.No. 15/594,518, which is incorporated herein by reference.

FIG. 3 illustrates a block diagram of a representative smart device 204in accordance with some implementations. In some implementations, thesmart device 204 (e.g., any devices of a smart-home environment 100,FIG. 1) includes one or more processing units (e.g., CPUs, ASICs, FPGAs,microprocessors, and the like) 302, one or more communication interfaces304, memory 306, communications module 342 with radios 340, and one ormore communication buses 308 for interconnecting these components(sometimes called a chipset). In some implementations, the userinterface 310 includes one or more output devices 312 that enablepresentation of media content, including one or more speakers and/or oneor more visual displays. In some implementations, the user interface 310also includes one or more input devices 314, including user interfacecomponents that facilitate user input such as a keyboard, a mouse, avoice-command input unit or microphone, a touch screen display, atouch-sensitive input pad, a gesture capturing camera, or other inputbuttons or controls. Furthermore, some smart devices 204 use amicrophone and voice recognition or a camera and gesture recognition tosupplement or replace the keyboard. In some implementations, the smartdevice 204 includes one or more image/video capture devices 318 (e.g.,cameras, video cameras, scanners, photo sensor units). The built-insensors 390 may include, for example, one or more thermal radiationsensors, ambient temperature sensors, humidity sensors, IR sensors,occupancy sensors (e.g., using RFID sensors), ambient light sensors,motion detectors, accelerometers, and/or gyroscopes.

The radios 340 enable one or more radio communication networks in thesmart-home environments, and allow a smart device 204 to communicatewith other devices. In some implementations, the radios 340 are capableof data communications using any of a variety of custom or standardwireless protocols (e.g., IEEE 802.15.4, Wi-Fi, ZigBee, 6LoWPAN, Thread,Z-Wave, Bluetooth Smart, ISA100.5A, WirelessHART, MiWi, etc.) custom orstandard wired protocols (e.g., Ethernet, HomePlug, etc.), and/or anyother suitable communication protocol, including communication protocolsnot yet developed as of the filing date of this document.

The communication interfaces 304 include, for example, hardware capableof data communications using any of a variety of custom or standardwireless protocols (e.g., IEEE 802.15.4, Wi-Fi, ZigBee, 6LoWPAN, Thread,Z-Wave, Bluetooth Smart, ISA100.5A, WirelessHART, MiWi, etc.) and/or anyof a variety of custom or standard wired protocols (e.g., Ethernet,HomePlug, etc.), or any other suitable communication protocol, includingcommunication protocols not yet developed as of the filing date of thisdocument.

The memory 306 includes high-speed random access memory, such as DRAM,SRAM, DDR RAM, or other random access solid state memory devices; and,optionally, includes non-volatile memory, such as one or more magneticdisk storage devices, one or more optical disk storage devices, one ormore flash memory devices, or one or more other non-volatile solid statestorage devices. The memory 306, or alternatively the non-volatilememory within the memory 306, includes a non-transitory computerreadable storage medium. In some implementations, the memory 306, or thenon-transitory computer readable storage medium of the memory 306,stores the following programs, modules, and data structures, or a subsetor superset thereof: operating logic 320 including procedures forhandling various basic system services and for performing hardwaredependent tasks; a device communication module 322 for connecting to andcommunicating with other network devices (e.g., network interface 160,such as a router that provides Internet connectivity, networked storagedevices, network routing devices, server system 164, etc.) connected toone or more networks 162 via one or more communication interfaces 304(wired or wireless); an input processing module 326 for detecting one ormore user inputs or interactions from the one or more input devices 314and interpreting the detected inputs or interactions; a user interfacemodule 328 for providing and displaying a user interface in whichsettings, captured data, and/or other data for one or more devices(e.g., the smart device 204, and/or other devices in smart-homeenvironment 100) can be configured and/or viewed; one or moreapplications 330 for execution by the smart device (e.g., games, socialnetwork applications, smart home applications, and/or other web ornon-web based applications) for controlling devices (e.g., executingcommands, sending commands, and/or configuring settings of the smartdevice 204 and/or other client/electronic devices), and for reviewingdata captured by devices (e.g., device status and settings, captureddata, or other information regarding the smart device 204 and/or otherclient/electronic devices); a device-side module 332, which providesdevice-side functionalities for device control, data processing and datareview, including but not limited to: a command receiving module 3320for receiving, forwarding, and/or executing instructions and controlcommands (e.g., from a client device 220, from a server system 164, fromuser inputs detected on the user interface 310, etc.) for operating thesmart device 204; a data processing module 3322 for processing datacaptured or received by one or more inputs (e.g., input devices 314,image/video capture devices 318, location detection device 316), sensors(e.g., built-in sensors 390), interfaces (e.g., communication interfaces304, radios 340), and/or other components of the smart device 204, andfor preparing and sending processed data to a device for review (e.g.,client devices 220 for review by a user); device data 334 storing dataassociated with devices (e.g., the smart device 204), including, but isnot limited to: account data 3340 storing information related to useraccounts loaded on the smart device 204, wherein such informationincludes cached login credentials, smart device identifiers (e.g., MACaddresses and UUIDs), user interface settings, display preferences,authentication tokens and tags, password keys, etc.; local data storagedatabase 3342 for selectively storing raw or processed data associatedwith the smart device 204 (e.g., video surveillance footage captured bya camera 118); a bypass module 336 for detecting whether radio(s) 340are transmitting signals via respective antennas coupled to the radio(s)340 and to accordingly couple radio(s) 340 to their respective antennaseither via a bypass line or an amplifier (e.g., a low noise amplifier);and a transmission access module 338 for granting or denyingtransmission access to one or more radio(s) 340 (e.g., based on detectedcontrol signals and transmission requests).

Each of the above identified elements may be stored in one or more ofthe previously mentioned memory devices, and corresponds to a set ofinstructions for performing a function described above. The aboveidentified modules or programs (i.e., sets of instructions) need not beimplemented as separate software programs, procedures, or modules, andthus various subsets of these modules may be combined or otherwiserearranged in various implementations. In some implementations, thememory 306, optionally, stores a subset of the modules and datastructures identified above. Furthermore, the memory 306, optionally,stores additional modules and data structures not described above.

FIGS. 4A-4B are perspective views of a representative camera assembly inaccordance with some implementations. FIG. 4A shows a first perspectiveview of a representative camera 118. As shown in FIG. 4A, the camera 118includes a head assembly 403, a stand assembly 402, and a cable 414(e.g., for powering the camera 118 and/or transferring data between thecamera 118 and a second electronic device.). The head assembly 403includes a cover element 404 and a casing 401 (also sometimes called ahousing). In accordance with some implementations, the cover element 404has IR transparent portions 412 for IR illuminators, visible and IRtransparent portion 416 for an image sensor, and semi-transparentportions 410 (corresponding to an ambient light sensor) and 408(corresponding to a status LED). In accordance with someimplementations, the cover element 404 also includes apertures 406 formicrophones. In accordance with some implementations, the casing 401includes an aperture 406-3 for a microphone.

In some implementations, the casing 401 has two or more layers. In someimplementations, the inner layer is composed of a thermally conductiveresin. In some implementations, the outer layer is a structural jacketconfigured to protect the camera 118 from environmental conditions suchas moisture or electromagnetic charge (e.g., static electricity). Insome implementations, the structural jacket is configured to protect thecamera 118 from impacts, such as from a collision with another object orthe ground.

FIG. 4B shows a back view of the camera 118. As shown in FIG. 4B thecable 414 is detachable from the stand assembly 402. For example, a usermay store the cable 414 separately from the camera 118 when desired. Inaccordance with some implementations, the casing 401 includes aplurality of apertures 417 for a speaker.

FIGS. 5A-5B are expanded component views of a representative cameraassembly in accordance with some implementations. The camera 118includes a cover element 404, an image sensor assembly 432, a speakerassembly 413, and a main circuit board 464. In some implementations, thespeaker assembly 413 includes a speaker and a heat sink. In someimplementations, the heat sink is configured to dissipate heat generatedat the main board 464. In some implementations, the speaker assembly 413acts as a heat sink for the camera's system-on-a-chip (SoC). In someimplementations, the SoC is thermally coupled to the speaker assembly413 with a thermal pad. In some implementations, the thermal pad's areais smaller than the speaker assembly's bottom surface 499. For optimalheat dissipation it is beneficial to spread the heat from the thermalpad over the entire bottom surface of the speaker assembly. In someimplementations, a thermally graphite sheet (e.g., thermally conductivesheet 466) is used to achieve this spreading since graphite has veryhigh in-plane thermal conductivity.

In some implementations, the camera 118 is a video streaming device withpowerful computing capability embedded in the device. Therefore, in someinstances, it will consume a lot of power and will also generate a lotof heat. In order to prevent the chipset and other components from beingdamaged by the heat, a thermal relief solution includes directing theheat from the CPU (e.g., a CPU of the SoC) to the speaker assembly 413.In some implementations, the speaker assembly 413 is composed of athermally conductive plastic that is structurally suitable and has goodheat spreading properties. In some implementations, a thermal pad on topof the shield can is used to direct the heat to the speaker assembly. Tofurther distribute the heat onto the speaker, in some implementations, agraphite sheet is placed on the bottom surface of the speaker assembly.In some implementations, the size of the graphite sheet is maximized toachieve the best thermal relief function.

The camera 118 includes the cover element 404 having the IR transparentportions 412 for IR illuminators, the apertures 406 for microphones, thesemi-transparent portion 408 corresponding to a status LED, and thesemi-transparent portion 410 corresponding to an ambient light sensor.The camera 118 also includes a plurality of heat pads 420 fordissipating heat from the main board 464 and a thermal receiverstructure 428 (e.g., having a shape like that of a fryer pot,hereinafter referred to as “fryer pot 428”) to the casing 401, aplurality of antennas 426 for wirelessly communicating with otherelectronic devices, a thermal mount structure 424 (e.g., having a shapelike that of a fryer basket, hereinafter referred to as “fryer basket424”) for dissipating and transferring heat from the image sensorassembly 432 to the cover element 404, and pads for thermally isolatingthe fryer basket 424 from the fryer pot 428.

In some implementations, the heat pads 420 are adapted to transfer heatfrom the fryer pot 428 to the casing 401. In some implementations, theheat pads 420 are adapted to thermally couple an inner layer of thecasing 401 and the fryer pot 428. In some implementations, the heat padsare composed of a plastic. In some implementations, the heat pads areadapted to thermally de-couple the fryer basket 424 from the fryer pot428. In some implementations, the fryer basket 424 is composed ofmagnesium. In some implementations, the fryer basket 424 is adapted todissipate heat from the image sensor assembly 432. In someimplementations, the fryer basket 424 is adapted to provide structuralsupport to the camera 118. In some implementations, the fryer basket 424is adapted to protect the image sensor assembly 432 from environmentalforces such as moisture and/or impact from objects and/or the ground.

In some implementations, the antennas 426 are configured to operateconcurrently using two distinct frequencies. In some implementations,the antennas 426 are configured to operate concurrently using twodistinct communication protocols. In some implementations, one or moreof the antennas 426 is configured for broadband communications (e.g.,Wi-Fi) and/or point-to-point communications (e.g., Bluetooth). In someimplementations, one or more of the antennas 426 is configured for meshnetworking communications (e.g., ZWave). In some implementations, afirst antenna 426 (e.g., antenna 426-1) is configured for 2.4 GHz Wi-Ficommunication and a second antenna 426 (e.g., antenna 426-2) isconfigured for 5 GHz Wi-Fi communication. In some implementations, afirst antenna 426 (e.g., antenna 426-1) is configured for 2.4 GHz Wi-Ficommunication and point-to-point communication, a second antenna 426(e.g., antenna 426-2) is configured for 5 GHz Wi-Fi communication andpoint-to-point communication, and a third antenna 426 (e.g., antenna426-3) is configured for mesh networking communication. In someimplementations, two or more of the antennas 426 are configured totransmit and/or receive data concurrently with others of the antennas426. MIMO (multi input multi output) provides the benefit of greaterthroughput and better range for the wireless communication.

One of the parameters in the antenna system is the isolation between twoantennas. Better isolation can ensure the data transmitted through twoantennas are uncorrelated which is the key to the MIMO system. One wayto achieve good isolation is to have large antenna separations. However,in modern consumer electronics the space left for antennas is very tightso having enough spacing between antennas is infeasible. While isolationis important, the antenna efficiency cannot be sacrificed. Isolation isdirectly related to how much energy is coupled from one antenna toanother. The Friis equation defines the power received by anotherantenna as inversely proportional to (1/R)², where R is the distancebetween two antennas. So increasing antenna spacing is one effective wayto achieve good isolation. Another means to achieve isolation is throughuse of a decoupling network. For example, an artificial coupling channelis generated in additional to its original coupling channel (e.g., whichis through air). By properly managing the two coupling channels, thegood isolation can be achieved.

In some implementations, the antennas 426 include at least one dual-bandInverted-F Antenna (IFA). In some implementations, the antennas are madeby FPC, LDS, Stamping, or other state of art antenna manufacturingtechnology. In some implementations, the fryer pot 428 is a systemground for one or more of the antennas 426. In some implementations, thesize of the antenna is about quarter-wavelength at 2.4 GHz. In someimplementations, each antenna includes a radiating element, a feed line,and a ground stub. The ground stub presents an inductance to compensatefor capacitance generated between the radiating element and the fryerpot 428. In some implementations, at least one of the antennas 426includes a second ground stub. The second ground stub is adapted tomatch the antenna to both 2.4 GHz and 5 GHz. In some implementations,the antenna feed is the feeding point for the 2.4 GHz and 5 GHz WiFisignal. In some implementations, the feed point is connected to theoutput of a WiFi chip. In some implementations, the antennas 426 includetwo identical IFA antennas. Both antennas are attached to the speakerassembly 413.

In some implementations, at least one of the antennas 426 includes asecond type of antenna having first radiating element, a secondradiating element, a first ground stub, and second ground stub. In someimplementations, the size of the first radiating element is aroundquarter wavelength of 5 GHz. In some implementations, the resonancefrequency at 2.4 GHz is determined by: (i) the size of the secondradiating element, (ii) the position of the first ground stub, and (iii)the position of the second ground stub. In some implementations, thefirst ground stub is placed at a pistol end of the second radiatingelement. In some implementations, the second ground stub is between thefirst radiating element and the first ground stub. In someimplementations, the position where second ground stub is attached tothe second radiating element is adjusted to tune to the resonantfrequency at 2.4 GHz. In some implementations, the first ground stub notonly acts as part of the antenna, but also a shielding element that canreduce coupling coming from the left-handed side of the first groundstub. In some implementations, the second ground stub is also ashielding element to further reduce the coupling coming from the lefthanded side of the antenna. In some implementations, the second type ofantenna includes more than 2 ground stubs. By using more ground stubsthe antenna's physical size can be enlarged while maintaining the sameresonant frequency (e.g., 2.4 GHz). In some implementations, the firstand second ground stubs are on the right-handed side of the firstradiating element to reduce coupling coming from the right-handed side.In some implementations, the antennas 426 include one or more antennasof a first type (e.g., IFAs) and one or more antennas of the secondtype.

By using a set of antennas including both a first type of antenna (e.g.,an IFA) and the second type of antenna, two antennas can be positionedin a tight space while maintaining both good efficiency and goodisolation between them. This enables the camera 118 to be compactwithout sacrificing the quality of wireless connectivity. In someimplementations, both types of antennas are manufactured by conventionalFPC technology with low cost. Unlike an antenna system relying on adecoupling system to achieve a similar isolation level, the IFA andsecond type antennas can be optimized and/or tuned independently.

The camera 118 may include the cover element 404, casing 401 withspeaker holes 417, the image sensor assembly 432, and a speaker assembly413. In some implementations, as shown, the speaker holes 417 extenddirectly outward from the speaker, which results in holes with anelliptical outer surface. In some implementations, the speaker holes 417are parallel to one another. In some implementations, the speaker holes417 extend outward at an angle consistent with the rear surface of thecasing 401 such that the holes have a circular, rather than elliptical,outer surface (not shown). The camera 118 also includes a light guide434 for directing light from a light assembly out the face of the camera118.

The camera 118 includes an infrared (IR) reflector 442, a light diffuser444, a light guide 446, a light ring 448, a microphone assembly 450, theimage sensor assembly 432, the fryer basket 424, stand coupling elements456 and 458, the fryer pot 428, a thermal insulator 462 adapted tothermally isolate the fryer pot 428 from the fryer basket 424, the mainboard 464, the thermally conductive sheet 466, the antennas 426, thespeaker assembly 413, and the casing 401. In accordance with someimplementations, the casing 401 has a lip 434 for reflecting anddirecting light from the light diffuser 444 outward from the face of thecamera 118.

In some implementations, the cover element 404 comprises achemically-strengthened glass. In some implementations, the coverelement 404 comprises a soda-lime glass.

In some implementations, the image sensor assembly 432 includes acircuit board (e.g., a PCB board), an IR cut filter, a lens holder, andan image sensor. In some implementations, the image sensor comprises a 4k image sensor. In some implementations, the image sensor comprises a 12megapixel sensor. In some implementations, the image sensor comprises awide-angle lens.

In some implementations, the thermally conductive sheet 466 is adaptedto dissipate heat generated by the main board 464 and/or transfer heatfrom the main board 464 to the speaker assembly 413 for subsequentdissipation outside of the camera via the rear portion of the casing401. In some implementations, the conductive sheet 466 is a graphitesheet. When a graphite sheet is placed near the antenna system withmultiple antennas, it can create a coupling medium between antennas. Theincreased coupling caused by the graphite can decrease the isolationbetween two antennas, thus degrading antenna efficiency or causingpermanent damage to the chipset.

In some implementations, the antennas 426 are configured to enable thecamera 118 to wirelessly communication with one or more other electronicdevices, such as a hub device 180, a smart device 204, and/or a serversystem 164.

In some implementations, the fryer pot 428 is composed of magnesium. Insome implementations, the fryer pot 428 is adapted to provide structuralsupport to the camera 118.

In some implementations, the fryer pot 428, the main board 464, theconductive sheet 466, the speaker assembly 413, and the antennas 426comprise a rear sub-assembly. Thermally de-coupling the fryer basket 424from the fryer pot 428 prevents heat generated by the main board 464from interfering with the image sensor assembly 432. In accordance withsome implementations, heat generated by the front of the main board 464is transferred to the fryer pot 428 to the heat pads 420 and dissipatedoutside of the camera via the casing 401 (e.g., the sides of thecasing). In accordance with some implementations, heat generated by theback of the main board 464 is transferred to the thermally conductivesheet 466 to the speaker assembly 413 and dissipated outside of thecamera via the back portion of the casing 401.

In some implementations, the rear sub-assembly is affixed to the casing401 via one or more fasteners (e.g., via 2-3 screws). In someimplementations, the cover element 404, the infrared reflector 442, thelight diffuser 444, the light guide 446, the light ring 448, and theimage sensor assembly 432 comprise a front sub-assembly. In someimplementations, the front sub-assembly is affixed to the casing 401 viaone or more fasteners (e.g., 2-3 screws). In some implementations, thefront sub-assembly is affixed to the rear sub-assembly via one or morefasteners.

In some implementations, the fryer basket 424 is adapted to dissipateheat generated by the image sensor assembly 432 and/or the light ring448. In some implementations, the fryer basket 424 includes one or moreforward-facing microphones. In some implementations, the downward-facingmicrophone 450 is operated in conjunction with the microphones on thefryer basket 424 to determine directionality and/or location of incomingsounds.

In some implementations, the IR reflector 442 is coated with an IRand/or visible light reflective coating. In some implementations, the IRreflector 442 is adapted to direct light from the IR illuminators 452 toa scene corresponding to a field of view of the image sensor assembly432.

In some implementations, the light ring 448 comprises a plurality ofvisible light illuminators (e.g., RGB LEDs), a plurality of IRilluminators 452, and circuitry for powering and/or operating thevisible light and/or IR illuminators. In some implementations, the lightguide 446 is adapted to direct light from the visible light illuminatorsout the face of the camera 118. In some implementations, the light guide446 is adapted to prevent light from the visible light illuminators fromentering the image sensor assembly 432. In some implementations, thelight guide 446 is adapted to spread the light from the visible lightilluminators in a substantially even manner. In some implementations,the light guide 446 is composed of a clear material. In someimplementations, the light guide 446 is composed of a poly-carbonitematerial. In some implementations, the light guide 446 has a pluralityof dimples to refract the light from the illuminators and prevent thelight from entering the image sensor assembly 432. In someimplementations, the light guide 446 is adapted to provide more uniformcolor and light output to a user from the illuminators. In someimplementations, the light guide 446 includes a plurality of segments,each segment corresponding to a visible light illuminator. In someimplementations, the light guide 446 includes one or more lightabsorbing elements (e.g., black stickers) arranged between each segmentto prevent light leakage from one illuminator segment to anotherilluminator segment.

In some implementations, the light diffuser 444 includes two or moresections (e.g., an inner section and an outer section). In someimplementations, the light diffuser 444 is adapted to diffuse the lightfrom the visible light illuminators. In some implementations, the lightdiffuser 444 is adapted to direct the light from the illuminators towardthe lip 434 of the casing 401. In some implementations, the light ring448 (and corresponding elements such as the light guide 446 and/or lightdiffuser 444) causes a circular colored (or white) light to be emittedfrom the front of the camera 118. In some implementations the componentsand corresponding light are circular and arranged around the peripheryof the front of the camera 118. They may encircle all or substantiallyall elements of the camera 118, such as the image sensor assembly 432,the IR illuminators 452, the ambient light sensor 451, a status LED, andthe microphone apertures 406. In other implementations, they arearranged not around the periphery but rather at an inner diameter, e.g.,around only the image sensor assembly 432. In yet other implementations,they do not surround any front-facing element of the camera 118. In someimplementations, they are arranged in a non-circular shape, such as asquare, oval, or polygonal shape. In some implementations, they are notarranged on the front of the device but rather a different surface ofthe device, such as the bottom, top, sides, or back. In someimplementations, multiple such light rings and components are arrangedonto the same or different surfaces of the camera 118.

The light ring 448 (and corresponding elements) may operate to indicatea status of the camera 118, another device within or outside of thesmart home environment 100 (e.g., another device communicatively coupledeither directly or indirectly to the camera 118), and/or the entireconnected smart home environment 100 (e.g., system status). The lightring 448 (and corresponding elements) may cause different colors and/oranimations to be displayed to a user that indicate such differentstatuses.

For example, in the context of communicating camera 118 status, when thecamera 118 is booting for the first time or after a factor reset, thering may pulse blue once at a slow speed. When the camera 118 is readyto begin setup, the ring may breathe blue continually. When the camera118 is connected to a remote cloud service and provisioning is complete(i.e., the camera is connected to a user's network and account), thering may pulse green once. When there is a service connection and/orprovisioning failure, the ring may blink yellow at a fast speed. Whenthe camera 118 is being operated to facilitate two-way talk (i.e., audiois captured from the audio and communicated to a remote device foroutput by that remote device simultaneously with audio being capturedfrom the remote device and communicated to the camera 118 for output bythe camera 118), the ring may breathe blue continuously at a fast speed.When the camera 118 is counting down final seconds before a factoryreset, the ring may close on itself at a rate equal to the time untilreset (e.g., five seconds). When the camera 118 has been factory andwhile the setting are being erased the ring may rotate bluecontinuously. When there is insufficient power for the camera 118 thering may blink red continuously at a slow speed. The visual indicationsare optionally communicated simultaneously, concurrently, or separatelyfrom audio indications that signal to the user a same or supplementalmessage. For example, when the camera 118 is connected to a remote cloudservice and provisioning is complete (i.e., the camera is connected to auser's network and account), the ring may pulse green once and output anaudio message that “remote cloud service and provisioning is complete.”

Additionally or alternatively, the camera 118 may communicate the statusof another device in communication with the camera 118. For example,when a hazard detector 104 detects smoke or fire sufficient to alarm,the camera 118 may output a light ring that pulses red continuously at afast speed. When a hazard detector 104 detects smoke or fire sufficientto warn a user but not alarm, the camera 118 may output a light ringthat pulses yellow a number of times. When a visitor engages a smartdoorbell 106 the camera 118 may output a light ring depending on theengagement; e.g., if the smart doorbell 106 detects motion, the camera118 may output a yellow light ring, if a user presses a call button onthe smart doorbell 106, the camera 118 may output a green light ring. Insome implementations, the camera 118 may be communicatively coupled tothe doorbell 106 to enable audio communication therebetween, in whichcase an animation and/or color of the light ring may change depending onwhether the user is speaking to the visitor or not through the camera118 or another device.

Additionally or alternatively, the camera 118 may communicate thecumulative status of a number of network-connected devices in the smarthome environment 100. For example, a smart alarm system 122 may includeproximity sensors, window break sensors, door movement detectors, etc. Awhole home state may be determined based on the status of such aplurality of sensors/detectors. For example, the whole home state may besecured (indicating the premises is secured and ready to alarm),alarming (indicating a determination that a break-in or emergencycondition exists), or somewhere in between such as pre-alarming(indicating a determination that a break-in or emergency condition mayexist soon or unless some condition is satisfied). For example, thecamera 118 light ring may pulse red continuously when the whole homestate is alarming, may pulse yellow when the whole home state ispre-alarming, and/or may be solid green when the whole home state issecured. In some implementations, such visual indications may becommunicated simultaneously (or separately from) with audio indicationsthat signal to the user the same or supplemental message. For example,when the whole home state is alarming, the ring may pulse red once andoutput an audio message that indicates the alarm “alarm”. In someimplementations, the audio message may provide supplemental informationthat cannot be conveyed via the light ring. For example, when the wholehome state is alarming due to a basement window being broken, the audiomessage may be “alarm—your basement window has been broken.” For anotherexample, when a pre-alarm amount of smoke has been detected by a hazarddetector 104 located in the kitchen, the audio message may be“warning—smoke is detected in your kitchen.”

In some implementations, the camera 118 may also or alternatively have astatus LED. Such a status LED may be used to less-instructivelycommunicate camera 118, other device, or multiple device statusinformation. For example, the status light may be solid green duringinitial setup, solid green when streaming video and/or audio datanormally, breathing green when someone is watching remotely, solid greenwhen someone is watching remotely and speaking through the camera 118,and off when the camera 118 is turned off or the status LED is disabled.It should be appreciated that the status LED may be displayedsimultaneously with the light ring. For example, the status LED may besolid green during setup while the light ring breathes blue, until theend of setup when the device is connected to the service andprovisioning is complete whereby the status LED may continue to be solidgreen while the light ring switches to a single pulse green.

Subject Monitoring Feedback Loop

The camera 118 described above in detail has many different uses in thesmart-home environment. In the context of a security system, the camera118 can detect human presence and/or motion, and can provide a real-timevideo feed of the monitored area to a user's smart phone or other mobilecomputing device. In a hazard-detection scenario, the camera 118 canprovide a real-time video feed of a situation in which a hazard mightexist. For example, if smoke is detected within the smart-homeenvironment, the camera 118 can provide a view to show areas of theenvironment that may be affected by the fire or smoke. The camera 118can be used to determine whether the alarm is a false alarm or an alarmsituation to which a response may be required. A single camera 118 orset of cameras can be installed in a smart-home environment, and theycan be put to many different simultaneous uses. For example, a singlecamera can be part of a home security system and part of a hazarddetection system at the same time. One of the many uses to which thecamera 118 can be simultaneously employed is that of monitoring anyinfant or other subject within the smart-home environment.

Many parents find comfort in being able to monitor their sleeping infantin real time. Video monitoring systems are available that provide a livevideo feed of an infant in their sleep environment. These live videofeeds are traditionally sent through an RF frequency communicationsystem to a dedicated video monitor or console that can be plugged in atdifferent locations in the user's home. However, these traditionalinfant monitoring systems that employ live video feeds suffer from anumber of drawbacks. First, these traditional systems typically employ alow resolution camera. The resultant video feed is typically grainy, andit is impossible to view details of the infant. Second, these camerasare typically unable to provide any additional information other thanthe live video feed itself. The live video feed provides very littleinformation on the health and/or safety condition of the infant.Moreover, because there is no interactivity in these traditional videofeeds, these baby monitors do not provide any meaningful emotionalconnection between the user and the infant.

In order to solve these and other technical problems, the embodimentsdescribed herein use the camera 118 with its high-resolution live videofeed in conjunction with many other smart-home devices to not onlymonitor the sleep of a subject, such as an infant or child, but to alsoemploy a feedback loop that optimizes the sleep conditions in thesmart-home environment. The plurality of smart-home devices monitors asleep environment to determine when the subject is in the sleepenvironment attempting to sleep. When the subject attempting to sleep isdetected, the smart-home system can transition into a sleep mode. Whilein the sleep mode, the smart-home devices may be placed in a mode suchthat they do not interrupt the sleep of the subject. For example,doorbells can be silenced, intercom systems can be switched off in thesleep environment, and other systems can be silenced or otherwise madeless likely to affect the sleep of the subject. While in the sleep mode,the smart-home devices can monitor the behavior of the subject to detecttimes when the sleep of the subject is interrupted. For example, thecamera 118 described above can detect motions (e.g., tossing, turning,rolling over, etc.) that indicate restless sleep. The camera 118 canalso detect temperatures of the subject that do not fit within apredefined thermal signature. The camera 118 in conjunction with othersmart-home devices can listen for audible sounds, such as coughing,sneezing, and crying, that indicate interrupted sleep patterns.

After detecting the interrupted sleep patterns, the smart-home systemcan employ a number of different methods to optimize the sleepenvironment and help put the subject back to sleep. When initiallyentering the sleep mode, the smart-home system can download a set ofdefault parameter values that govern the operation of the smart-homedevices in and/or around the sleep environment. These parameters mayinclude thermostat temperatures, thermal signatures of the subject, airfilter operation, noise generation, lighting, music, and/or otherenvironmental stimuli. When the subject's sleep patterns areinterrupted, the behavior indicating the interrupted sleep can bematched to one or more of these environment or stimuli. For example, ifthe subject is coughing, the smart-home system can increase theoperation of a humidifier and/or air filter. When the subject becomestoo hot/cold, the smart-home system can decrease/increase the setpointtemperature of the thermostat in the sleep environment. In addition tolinking interrupted sleep behavior to specific smart-home system controladjustments, additional sleep-aid methods can be employed, such aswhite-noise generation, low-level lighting, music, vibrations, and othersmart-home actions that can help put the subject back to sleep. If thesubject stays awake, the system can send an alert to a computing deviceof a parent or other monitoring individual and can transition out of thesleep state into a normal operating state.

The sleep behavior of the subject can be monitored over time and canaffect the default parameter values of the smart-home devices while inthe sleep state using a closed-loop feedback system. The defaultparameter values can initially be determined and downloaded from aserver that monitors/interacts with the smart home devices. Thesedefault values can be based on a population of similar subjects, and canrepresent parameters that have produced an optimal sleep environmentwithin the population of similar subjects. For the particular subjectbeing monitored in the smart-home environment, these values may not beinitially optimal. As interrupted sleep is detected and remedied bychanging the parameters of the smart-home devices, these changes can befed into a neural network comprised of machine learning algorithms thatdetermine when the changed parameter should permanently affect thedefault parameters that are assigned to the smart-home devices in thatparticular smart-home environment when entering the sleep state.Furthermore, as the parameters for an individual subject/environment arechanged, these changes may also be provided to the server. When asufficient number of similar changes for a similar subject populationare received, the default values provided by the server may also bechanged for that population using neural network. Therefore, theclose-loop feedback system can operate at an individual level and/or apopulation level for a plurality of smart-home environments.

Throughout this disclosure, the monitoring system may use the monitoringof an infant or child as an example. However, other embodiments are notlimited to an infant in a sleep environment. Some embodiments maymonitor other types of subjects, such as the elderly, the physicallydisabled, and/or other human subjects. These subjects may also bemonitored in any environment aside from a sleep environment. Forexample, a subject could be monitored in a wheelchair, in a swimmingpool, in a bed, in a recliner, on a couch, and/or any other environment.Although the camera 118 described above is also used as an example,other embodiments may use different home video cameras that areconfigured to capture a live and/or thermal video feed of the monitoredsubject. Any smart-home device may also be used as part of thesmart-home system to monitor the sleep of the subject and/or provideenvironmental controls and/or stimuli to help improve/optimize the sleepenvironment. The hazard detector, home assistant, thermostat, airfilter, camera, and/or lighting systems described herein are provided asexamples, and are not meant to be limiting. Other smart-home systems mayalso be incorporated into the smart-home environment to monitor/aid thesleep of the subject.

FIG. 6 illustrates an infant 602 sleeping in a sleep environment 604 andbeing monitored by a plurality of smart-home devices, according to someembodiments. The plurality of smart-home devices may include a camera118. The camera 118 may be one of many video cameras that aredistributed throughout the smart-home environment. Although not depictedexplicitly in FIG. 6, the sleep environment 604 may also includeadditional cameras, some of which may also be positioned to observe theinfant 602. Other cameras may be used for security purposes or may beconfigured to observe the infant 602 in other positions and/or otherlocations within the sleep environment 604. In some embodiments, thecamera 118 may include both a regular, visible-light camera functionalong with a thermal imager. The thermal imager may include anon-contact temperature measurement device that records thermal energyat pixel locations. The thermal imager can detect infrared energy thatis emitted and/or reflected by any object or subject within itsfield-of-view. The thermal imager can convert the thermal emissions intoa thermogram, or thermal image, that can be displayed and/or analyzed bya computing device. The thermal imager need not be as high resolution asthe camera 118. For example, some embodiments may use a thermal imagerhaving a 100×100 pixel resolution. This low resolution may be sufficientfor detecting thermal conditions sufficiently to diagnose variousproblems that may be associated with the subject being monitored. Usinga low resolution thermal imager can also reduce the amount of memory,processing power, and/or bandwidth required for the analysis describebelow. High-resolution thermal imagers can be used to detect smallerproblems at greater distances, and the final resolution of the thermalimager on the camera 118 may be selected based on a distance of thecamera 118 from the infant 602.

In this example, the camera 118 is positioned within the sleepenvironment 604 such that a live video feed of the infant 602 can becaptured. Some embodiments may include automatic pan/tilt mounts thatuse computer vision algorithms to automatically train the camera 118 onthe infant 602 such that the infant 602 can be located anywhere in thesleep environment 604 and still be monitored by the camera 118. In someembodiments, the pan/tilt mounts can be driven by motion detectionalgorithms in the camera, such that they focus on motion detected intheir field-of-view. Some embodiments may use one or more cameras thatare trained specifically on different locations within the sleepenvironment 604, such as the crib 606, the floor, a changing table,and/or the like. Each of these cameras may be activated when movement isdetected within their field of view. Therefore, when the infant 602 isplaced in the crib 606, the camera 118 can automatically detect theshape and/or movement of the infant 602 and determine that the infant602 is within its field of view. Camera 118 can then automatically beginrecording, analyzing, and/or transmitting a live video feed of theinfant 602. Some embodiments may require a facial image of the infant602 to detect fevers and other heat-related conditions. Theseembodiments may use known facial recognition algorithms to locate theface of the infant 602 within its field-of-view and pan/tilt/zoom thecamera 118 accordingly to center its field-of-view on the face of theinfant 602.

The sleep environment 604 may also include additional smart-homedevices. Some embodiments may include a hazard detector 104. The hazarddetector 104 may include a low-level light ring that can provide a smallamount of light to the sleep environment 604. The hazard detector 104may also include a speaker that can provide sound, such as white noise,music tracks, ocean sounds, and so forth. The hazard detector 104 may beconnected via a smart-home network to the other smart-home devices. Someembodiments may also include a thermostat 102. The thermostat 102 mayinclude an active electronic display that can provide a small amount oflight to the sleep environment 604. The thermostat 102 can also managethe temperature of the sleep environment 604 by measuring the ambienttemperature in the sleep environment 604 and comparing the measuredambient temperature to a target setpoint temperature. Some embodimentsmay also include a home assistant 610. The home assistant may be avoice-activated, network-connected speaker. The home assistant 610 canprovide low-level lighting and can play a variety of sounds in the sleepenvironment 604. For example, the home assistant 610 can play music,white noise, prerecorded messages or songs from a parent, and/or thelike. The home assistant 610 may also include a microphone that canrecord and/or transmit any sound that is generated in the sleepenvironment 604. In some embodiments, the home assistant 610 may becapable of processing sounds made by the infant 602 and generating audiooutputs based on those sounds and/or sending an alert to a monitoringdevice outside the sleep environment 604. Some embodiments mayadditionally include an air purifier 608. The air purifier 608 canprovide a low-level noise in the sleep environment 604 and can filterand/or circulate air around the sleep environment 604. The smart-homedevices described above are only provided by way of example and are notmeant to be limiting. Additional smart-home devices may be present inthe sleep environment 604, such as motion detectors, smart outlets,controlled lighting, intercoms, video monitors, and so forth.

The sleep environment 604 may include a bedroom, a closet, a nook,and/or any other location within the smart-home environment. The sleepenvironment may be characterized in that it includes a bed, a crib 606,a couch, a sofa, a porta-crib, a mattress, a sleeping pad, an airmattress, a covered section of the floor, and/or any othersleep-suitable location. Although the infant 602 is depicted in FIG. 6,any other monitored subject may also be monitored by the camera 118 andused in conjunction with the algorithms described in detail below. Othersubjects may be monitored in sleep environments and/or any other type ofmonitored environment within the smart-home environment.

The smart-home devices depicted in FIG. 6 can work together inconjunction with other smart-home devices in the home to monitor andcontrol the sleep environment of the monitored subject, such as theinfant 602. FIG. 7 illustrates a flowchart 700 of a method fordetermining when the smart-home devices in the home should enter into asleep mode, according to some embodiments. The method may includeoperating in a normal/awake mode (702). The normal mode of operation mayallow the smart home devices within the home to operate according totheir normal schedules and/or inputs. For example, a stored setpointschedule may be used by the thermostat 102 to regulate a temperaturethroughout the home and in the sleep environment 604. An alarm systemmay be free to trigger an audible alert when it detects an unauthorizedpresence. An intercom system may allow users to communicate with anyroom in the home. The hazard detector 104 may be free to sound an alarmwhen smoke is detected, and so forth. The normal mode of operation maydescribe operations where no changes are made to the operation of anysmart-home devices based on optimizing the conditions of the sleepenvironment 604.

The method may also include monitoring the sleep environment foractivity (704). In some embodiments, the sleep environment can bemonitored for activity that indicates the presence of the subject in thelocation performing actions that indicate that the subject should beasleep. For example, the camera 118 can use computer vision algorithmsto recognize that a human subject has entered the sleep environment 604and is in the crib 606. Motion detectors in the sleep environment 604that are part of a security system can detect a motion pattern thatindicates someone placing the infant 602 in the crib 606. Each of thesesystems can individually and/or collectively detect the presence of theinfant 602 in the correct sleep location, such as the crib 606.

As the sleep environment 604 is monitored by the smart-home devices, adetermination can be made as to whether or not a child is detected(706). Some embodiments may detect more than simply the presence of theinfant 602. For example, these embodiments may determine the presence ofan infant 602 that is asleep based on motion detection algorithmsexecuted by the camera 118. These embodiments may also determine thepresence of an infant 602 with a facial signature that is consistentwith a predetermined facial signature of the infant 602 when it isasleep. Some embodiments may recognize the presence of the infant usingthe camera 118 and/or other motion/presence sensors to determine thatthe infant 602 is present in the crib 606 and not making any noise usingthe microphone of the home assistant 610 and/or any of the othersmart-home devices. Therefore, one or more of the smart-home devices,acting individually or collectively, can process inputs in the sleepenvironment 604 to determine that the infant 602 is present in the sleepenvironment 604 for the purpose of sleeping. When such a determinationis made, the smart-home system can enter into a sleep mode (710) asdescribed in detail below.

Some embodiments may additionally or alternatively make a seconddetermination based on user inputs (708). In some embodiments, users canprovide an input to one or more smart-the home devices indicating thatthe system should enter the sleep mode. For example, a user can tell thehome assistant 610 that the infant 602 is asleep and ask the homeassistant 610 to place the smart-home system into the sleep mode for aspecified time interval. In some embodiments, users may provide a sleepschedule to the smart-home system indicating that the infant 602 willsleep between 2:00 PM and 4:00 PM each day. This input can cause thesystem to automatically enter the sleep mode during the specified timeinterval. In some embodiments, users may provide a sleep schedule thatindicates a start time for a scheduled nap/bedtime, and allow thesmart-home system to monitor the sleep pattern of the infant 602 andautomatically transition out of the sleep mode when the sleep of theinfant 602 is interrupted for a predetermined time interval. Someembodiments may automatically learn a sleep schedule of the infant 602over time using inputs received by the camera 118 and other smart-homedevices in the sleep environment 604. The smart-home system can thenautomatically generate a sleep schedule that is learned based on thebehavior of the infant 602 and/or the parents of the infant 602. Thesmart-home system can then implement a sleep schedule based on theobserved behavior of these human occupants. Some embodiments may use acombination of user inputs, learned or provided sleep schedules, and/orimmediate smart-home device sensor inputs. For example, the smart-homedevices may use thresholds to determine when the infant 602 is presentand sleeping. These thresholds may be lowered during time intervals thatcoincide with a predetermined and/or learned sleep schedule. User inputsindicating that the system should enter the sleep mode may override anyautomatic determinations made by the smart-home system based on sensorinputs and/or stored schedules.

When the situations described above determine that the system shouldenter the sleep mode (710), certain smart-home device parameters can bealtered throughout the home. The sleep mode may be intended to preventthe smart-home system from interrupting the sleep of the infant 602without sufficient cause, such as an emergency situation. Therefore,operating parameters of smart-home devices inside and/or outside thesleep environment 604 can be changed temporarily during the sleep mode.For example, the hazard detector 104 may be prevented from broadcastingaudible warnings in the sleep environment 604 for a short time intervalwhen smoke is detected in other parts of the home away from the sleepenvironment, such as the kitchen. An intercom system may be preventedfrom generating sound in the sleep environment 604 without a specificcommand indicating a user intends to broadcast into the sleepenvironment 604. Any noise-generating smart-home devices can have theirsound outputs lowered and/or disabled when in the sleep mode. Forexample, a doorbell system can be lowered in volume, disabled, and/oronly allowed to generate noise on a level/floor of the home that isdifferent from the sleep environment 604. It will be understood that anyparameter change in the sleep mode need not disable or lessen theeffectiveness of any emergency systems. Hazard detectors throughout thehome can still detect smoke and warn occupants of danger. Securitysystems may still detect intruders and unauthorized entry. Generally,users may provide preferences that define alert methods that can stillbe effective while in the sleep mode without generating loud noises inthe sleep environment 604.

In addition to applying the default sleep mode parameters for thesmart-home devices when entering the sleep mode (710), some embodimentsmay also activate one or more sleep aids (712). Activating the sleepaids will be described in detail below in relation to FIG. 12.Generally, activating sleep aids may include turning on low-levellighting, playing white noise, lullabies, voice recordings, and/or othercomforting sounds that may help induce sleep, activating moving devicessuch as crib-mounted mobiles or crib-vibration devices, and so forth.The sleep aids can be activated automatically when entering the sleepmode to help the infant 602 quickly go to sleep. The system can monitorthe behavior of the infant 602 using the camera 118 and other smart-homedevices to determine when the infant 602 is asleep. Once the infant 602is asleep, the sleep aids can be deactivated. The sleep aids can bedeactivated one-at-a-time and/or may be gradually turned down or turnedoff such that deactivating the sleep aids is not abrupt. In someembodiments, the sleep aids can be turned on when entering the sleepmode for a predetermined time interval, such as 15 minutes, then turnoff automatically at the end of the predetermined time interval.

FIG. 8 illustrates a view of the infant 602 that may be received by thecamera 118, according to some embodiments. In many situations, theinfant 602 may be relatively immobile while sleeping. For example, theinfant 602 may be confined to the crib 606 while sleeping. Therefore,the field of view of the camera 118 may be adjusted or shrunk to capturein high resolution only the area in which the infant 602 may occupy inthe sleep environment. This may include automatically panning/tiltingthe camera to center its field of view on the infant 602. Additionallyand/or alternatively, a zoom of the camera 118 can be adjusted such thatthe infant 602 substantially fills the field-of-view of the camera 118.For example, the zoom of the camera may be adjusted until the infant 602fills approximately 50% of the field of view of the camera 118. In someembodiments, the automatic pan/tilt of the camera 118 can be influencedby a facial recognition algorithm that recognizes the face of the infant702 in its field-of-view and causes the camera to center itsfield-of-view on the face of the infant 602. The camera 118 described indetail above includes, for example, a 4 k resolution. In someembodiments, the live video feed of the camera 118 may be one of theprimary ways of monitoring the sleep of the infant 602.

FIG. 9 illustrates a camera view and of the infant 602 with a boundingbox 902 that reduces the processing power, memory, and/or bandwidthrequired by the system, according to some embodiments. Some embodimentsmay analyze the real-time video feed of the infant 602 at the camera118. In these embodiments, it may be advantageous to decrease the numberof pixels that need to be analyzed in the full field-of-view of thecamera 118. By reducing the analyzed portion of the field-of-view of thecamera 118, algorithms can run faster and less memory can be used on thecamera 118. This may be particularly advantageous for cameras that haverelatively small processing capabilities and/or relatively limitedmemory storage available.

In some embodiments, the video feed of the camera can be analyzed inreal time to identify a portion of each image frame that includes theface of the infant 602. A bounding box 902 can be selected that includesthe face, along with a predetermined amount of each surrounding image.For example, a bounding boxing 902 can include the face at the center ofthe bounding box 902, and can also expand to include an additional twofeet of image extending outward from the face. In other embodiments, thebounding box 902 can include the face, as well as a surrounding areathat can be visually identified as a subject (e.g. the infant 602). Inthe example of FIG. 9, the bounding box 902 includes the face as well asthe rest of the upper body of the infant 602. Computer vision algorithmscan be used to identify objects, and these computer vision algorithmscan be modified based on this particular environment to identify theshape of the infant 602 in each frame. The bounding boxing 902 can besized such that the entire infant 602 is captured within the analyzedarea for the camera 118.

In some embodiments, the resolution of the captured video feed can bealtered based on the bounding box 902. For example, the camera canrecord and transmit a lower resolution video image for portions of theimage outside of the bounding box 902, while preserving ahigh-resolution digital image for portions of the video feed that areinside the bounding box 902. This can simplify the video processing of athermal-signature-matching algorithm, decrease the bandwidth required totransmit the live video feed in real time, reduce the amount ofprocessing power required to detect the small motions 702 in the livevideo feed, and/or reduce the amount of memory required to store imagesand information associated with the live/thermal video feeds.

As described above, the live video feed of the camera 118 can be one ofthe primary inputs used to determine that the infant 602 is present inthe sleep environment 604 and in a sleep situation such that the sleepmode of the smart-home system should be entered. As described above, thecamera 118 can capture a view of the infant 602 and use computer visionalgorithms, motion detection algorithms, thermal signatures, noisedetection, object/facial recognition algorithms, and so forth todetermine the presence of the infant 602 in a sleep situation. In someembodiments, the camera 118 can distinguish between situations where ahuman other than the infant 602 is present in the sleep environment 604.For example, when a parent is cleaning the sleep environment 604, thesmart-home system may distinguish between the parent and the infant 602based on the size of the parent, the thermal signature of the parent,the facial recognition of the parent, the motion patterns of the parent,and so forth. Additionally, the camera 118 can distinguish betweensituations where the infant 602 is present in the sleep environment 604for the purpose of sleeping and situations where the infant 602 ismerely present in the sleep environment 604. For example, if the infant602 is crawling on the floor of the sleep environment 604 and not in thecrib 606, it is unlikely that the infant 602 is in the sleep environment604 for the purpose of sleeping. In another example, if the infant 602is standing in the crib 606, then it is possible that the infant 602 isin the crib 606 to play, for a timeout, to be kept off the floor of thesleep environment 604 while the parent cleans, and so forth. Generally,a parent can set preferences in the smart-home system that define thesituations in which the infant 602 is present in the sleep environment604 for the purpose of sleeping. In some embodiments, these preferencescan be set automatically as the smart-home system learns the behavior ofthe infant 602. For example, if the camera 118 observes the infant 602lying in the crib 606 every day from 2:00 PM to 4:00 PM, the smart-homesystem can use these characteristics (i.e., infant in the crib, infantlying down, etc.) to define behaviors and/or times when the systemshould enter the sleep mode.

FIG. 10 illustrates a system for providing default parameters forsmart-home devices, according to some embodiments. The plurality ofsmart-home devices 1006 each may operate using one or more storedparameters. The thermostat 102 may use stored setpoints and/or setpointschedules as parameters that control the operation of the HVAC system.The air purifier 608 may include an operating schedule and/or specifiedlevel of air purity that governs its operation. Various lighting systemsin the sleep environment, including light provided by the hazarddetector 104 and/or the home assistant 610, may include light levelsand/or schedules defining how light should be provided. Any smart-homedevices having speakers or noise generation capabilities may includeparameters that define times, volume, and sound content for any soundoutput. Each of these parameters may be provided by user inputs, defaultparameters, and/or learning algorithms in the smart-home environment.

When transitioning to the sleep mode, new parameters may be provided toany of the smart-home devices in the home as described above. These newparameters may be configured to optimize the conditions of the sleepenvironment 604 to be conducive to the sleep of the infant 602 whilepreventing any external sleep interruptions that can be avoided. In someembodiments, these default parameters may be preprogrammed into each ofthe smart-home devices in the home in a manufacturing orpost-manufacturing process. In some embodiments, these defaultparameters may be provided through inputs received from users during asetup process through, for example, a progression of interview-styleuser interfaces. In some embodiments, these default parameters may belearned through user inputs during operation of the various smart-homedevices. For example, if users consistently increase the temperature inthe sleep environment and cause the home assistant 610 to provide whitenoise between 2:00 PM and 4:00 PM, the smart-home system can use theseuser changes as parameters for the sleep mode rather than incorporatingthem as changes to the normal setpoint schedules for the smart-homedevices during the normal operating mode.

In some embodiments, a server 164 that monitors one or more of thesmart-home devices may provide the default parameters for the sleepmode. The server 164 may be configured to monitor smart-home devices inmany different homes across a wide variety of geographic locations. Theserver 164 can generate profiles for different types of subjects invarious sleep environments. For example, the server 164 can clusterdifferent sleep environments based on characteristics of the subjectsand/or the sleep environments themselves and generate default parameters1004 for the population of subjects in each cluster. In someembodiments, populations of subjects can be clustered together based onsubject characteristics, such as length, weight, age, gender, birthmethod, nursing status, teething status, and/or any other physicalcondition that contributes to a sleep pattern of the subject.Populations of subjects may also be clustered together based oncharacteristics of the sleep environment, such as room size, sleep-area(e.g., crib) size, home square footage, sound/temperature insulation,architectural layout, smart-home devices present, number of occupants inthe home, external noise sources, geographic location, ZIP Code,altitude, weather, and so forth.

Before entering the sleep mode, the smart-home system can send a requestto the server 164 requesting the default parameters 1004 that arespecifically tailored for a population that is similar to the infant 602and/or similar to the sleep environment 604. In order for the server 164to select the corresponding default parameters 1004, users can elect toprovide any subject characteristics 1002 to the server 164. Whenpermission is granted to send the subject characteristics 1002 to theserver 164, the server 164 can identify a population cluster into whichthe infant 602 and/or sleep environment 604 would be placed.

The default parameters 1004 for that cluster can then be sent back tothe smart home devices 1006 for use during the sleep mode.

As will be described in greater detail below, the default parameters1004 can act as an initial setting in the local smart-home environment.A closed feedback loop can be used locally to optimize those defaultparameters 1004 such that they are tailored for the specific situationof the infant 602 rather than just a population that is similar to theinfant 602. By using the default parameters 1004 as a starting point,users are relieved of the need to determine their own initial settingsfor the sleep mode, which can be a very technical process. Users mayalso not be sure how to optimize each of the smart home devices in thesleep environment 604. Additionally, providing default parameters 1004that are close to the optimal parameters for the particular sleepenvironment 604 may allow the system to optimize those parameters fasterand converge towards the optimal sleep environment far more rapidly thanit could if starting with parameters that were not derived from asubstantially similar population and/or environment.

FIG. 11 illustrates a flowchart 1100 of a method for monitoring thesleep of the subject and adjusting environmental conditions to optimizethe sleep environment, according to some embodiments. The method mayinclude entering the sleep mode (1102). The sleep mode may be enteredusing any of the methods described above. The method may also includemonitoring the behavior of the subject (1104). Any of the smart-homedevices in the sleep environment 604 can be used individually and/or incombination to monitor the behavior of the subject. In some embodiments,the subject may be specifically monitored to determine whether thesubject is sleeping soundly or whether the sleep of the subject is beinginterrupted (1106).

In some embodiments, the motion of the subject can be monitored by thecamera 118. A motion detection algorithm can be used to determine if thesubject is moving more than a threshold amount while the subject issupposed to be asleep. The threshold can be set such that small and/orisolated motions that do not generally indicate interrupted or restlesssleep can be ignored. Computer vision algorithms can identify individualbody parts of the infant 602, and compare the motion of the infant toknown motion patterns that indicate interrupted sleep. For example, aninfant moving their arms up-and-down or kicking their legs repeatedlymay indicate interrupted sleep. These motion patterns can be detected bythe camera 118 to determine that the sleep of the infant is interrupted.In another example, an infant attempting to roll over or rolling overrepeatedly may indicate interrupted sleep. Again, this motion patternmay be detected by the camera 118 to determine that the sleep of theinfant is interrupted. In another example, relatively small, isolatedmovements of the arms/legs of the infant may simply indicate normalmotions that are experienced by the infant 602 during sleep, andtherefore need not indicate restless sleep. Because the duration ofthese motions and/or the magnitude of these motions would fall below thepredetermined threshold, the system can determine that these motions donot necessarily indicate interrupted sleep unless they continue in timeand/or increase in magnitude.

In addition to detecting motion, motion patterns, and/or motionsignatures, the camera 118 can also be used to detect other behaviors orcharacteristics of the infant 602 in order to determine when sleep isbeing interrupted. For example, the camera 118 may include a thermalimaging camera that can provide a thermal image of the infant 602. Aswill be described in greater detail below. The thermal image of theinfant 602 can be analyzed to detect characteristics of the infant, suchas being too cold, being too hot, experiencing medical problemsincluding fever, teething, and/or infection, and so forth. The camera118 can also capture and analyze the skin color of the infant 602. Askin color that is too pale or blue can indicate poor circulation and/ora sleep environment 604 that is too cold. Skin that is red and/orflushed may indicate a sleep environment 604 that is too hot, as well asa medical conditions such as a fever. A baseline skin color for theinfant 602 during normal sleep can be established during a traininginterval, and that baseline skin color can be compared to a current skincolor by the camera 118 during future sleep intervals. Deviation fromthe baseline skin color can be used to determine that the sleep of theinfant 602 is not optimal, and can cause the smart-home system togenerate a response as described below.

The camera 118 can also detect additional situations where the sleep ofthe infant may be interrupted. In some embodiments, a facial expressionof the infant 602 can be compared to a baseline facial expression thatis learned by the smart-home system while the infant 602 is sleeping.The facial expression can be captured using the visible light imager ofthe camera 118 and/or the thermal imager of the camera 118. Facialrecognition algorithms can be used to not only identify the infant 602,but can also be used to compare a baseline facial expression to acurrent facial expression captured by the camera 118. If changes infacial expression continue for more than a threshold amount of time ordeviate from the baseline facial expression by more than a thresholdamount, the smart-home system can determine that the sleep of the infant602 is being interrupted.

In some embodiments, sensors on smart-home devices in the sleepenvironment 604 can monitor the ambient conditions in the sleepenvironment 604 and determine if a significant deviation from thedefault parameters occurs. For example, the thermostat can monitor thetemperature in the sleep environment 604 and determine whether theambient temperature in the room is staying near enough to a targettemperature specified in the default parameters. If the room is notmaintaining its temperature as it should, this can indicate poorinsulation or even an open window. If the air purifier 608 is filteringan abnormal amount of particles from the air of the sleep environment604, this can indicate that something is causing the air quality to bediminished. If the hazard detector 104 detects smoke, this can indicatea dangerous situation in the sleep environment 604. Each of these typesof indications may typically coincide with the sleep of the infant 602being interrupted. Therefore, these environmental abnormalities can alsobe used as indications that the sleep of the infant 602 is beinginterrupted or may soon be interrupted. Additionally, in cases whereenvironmental abnormalities in the environment are detected, thresholdsfor the actual behavior of the infant 602 can be lowered. For example,if the temperature for the sleep environment 604 is below the setpointtemperature, then the thermal threshold for the thermal image of theface of the infant 602 can be lowered to determine that sleep is beinginterrupted sooner. In another example, if the air quality in the sleepenvironment 604 is below a threshold quality, then a sound threshold forduration of magnitude for detected coughing/sneezing can be lowered todetermine that the sleep of the infant 602 is being interrupted sooner.

In some embodiments, the behavior of the subject can be monitoredthrough sound using microphones on various smart-home devices. As withmotion detection, small, isolated sounds that fall below a threshold inmagnitude and/or duration can be ignored by some embodiments. Otherembodiments may determine that any sound made by the infant isindicative of interrupted sleep. Different types of interrupted sleepcan be determined based on an audio signature of any sound recorded fromthe infant. These audio signatures can be downloaded from a databasefrom the server 164 in a manner similar to how the default parameters1004 were downloaded such that these audio signatures can be fit to apopulation of subjects that are similar to the infant. These audiosignatures may include signatures for crying, sneezing, coughing,gagging, sucking on a pacifier, wheezing, snoring, and/or any othernoise that may indicate restless sleep in a subject. For subjects thatare becoming verbal, the smart-home system can learn over time differentaudio sounds made by the particular subject and determine whether theyindicate interrupted sleep or normal sounds made during normal sleepcycle. For example, some children may talk in their sleep or routinelymake other noises that do not indicate that their sleep is restless orbeing interrupted. Similarly, some children may make specific sounds orsay words/phrases when they are waking up. Over time, the smart-homesystem can record these different sound types and classify them assounds that indicate interrupted sleep, normal sleep, or an awake statefor the subject.

When the sleep of the infant 602 is determined to be interrupted, thesmart-home system can activate one or more sleep aids and/or adjust thephysical conditions of the sleep environment 604. The algorithm fordetermining which sleep aids to activate and/or which environmentalconditions to adjust is described in detail below in relation to FIG.12. Generally, behaviors of the infant 602 can be linked to specificenvironmental conditions. The smart-home devices can then adjust thedefault parameters governing these environmental conditions tospecifically address the behaviors of the infant 602. A plurality ofsleep aids (e.g., night-lighting, white noise, vibrations, etc.) thatare not necessarily linked to aberrant environmental conditions can alsobe employed to help automatically soothe the infant 602 and put themback to sleep.

The system can continue employing sleep aids and/or adjustingenvironmental conditions to help soothe the infant 602 when their sleepis interrupted. However, at some point it may be determined that theinfant 602 is fully awake and not going back to sleep. A determinationcan be made as to whether the interrupted sleep has continued longenough to assume that the infant 602 will not go back to sleep on theirown (1110). For example, if the crying of the infant 602 exceeds apredetermined time interval or exceeds an intensity/volume threshold, adetermination can be made that the infant 602 may not go back to sleepon their own. In another example, if the infant 602 rolls onto theirstomach, the smart-home system can transition out of the sleep mode andalert a monitoring device accessible by a parent or guardian. In someembodiments, the smart-home system can continuously learn differentbehaviors of the subject to determine when the system should transitionback to the normal operating mode (1112). For example, if the infant 602gets too cold as detected by the camera 118, the thermostat 102 canadjust the temperature in the sleep environment 604 accordingly.However, once too cold, the infant 602 may not go back to sleep on theirown. After this situation occurs multiple times, the smart home systemcan learn that when the temperature of the infant 602 gets too cold andthe infant 602 cries for more than two minutes, the system shouldtransition back to the normal operating mode and alert a parent. Thus, aclosed-loop feedback system can continuously update a set of conditionsand/or behaviors that indicate that the child is awake enough that thesystem should no longer operate in the sleep mode.

FIG. 12 illustrates a flowchart 1200 of a method for automaticallysoothing a subject whose sleep has been interrupted, according to someembodiments. As described above, the method may include causing thesystem to enter the sleep mode (1202) and monitoring the behavior of thesubject (1204) to determine when the infant's sleep is interrupted(1206). When the child sleep is interrupted or may be interrupted, thebehavior of the child indicating that the sleep is interrupted can oftenbe associated directly with environmental condition that can be adjustedto suit that specific behavior (1208). Thus, the smart-home systemattempts to identify a cause-and-effect relationship between specificbehaviors of the infant 602 and specific environmental conditions thatcan be adjusted by the smart-home system.

For example, one such relationship between environmental conditions andbehaviors can be associated with air quality. The air purifier 608 canuse default parameters that are “baby safe” to help parents know thatthe air in the sleep environment 604 is safe to breathe. For example,the carbon monoxide threshold alert level can be based on infant age andlevel of exposure to make the room more safe for the infant 602. The airpurifier 608 can also monitor air quality, mold, volatile organiccompounds (VOCs) and dangerous ultrafine particulates. The smart-homesystem can track these levels over time, and alert parents when theyrise to harmful levels. The smart-home system can also adjust theoperation of the air purifier 608 to bring these levels back to a “babysafe” condition. The smart-home system can also provide alerts toparents and provide recommendations on actions that can be taken tobring these air quality levels into an acceptable range.

In some embodiments, when coughing, sneezing, wheezing, laboredbreathing, or breathing through a stuffy nose are detected as possiblyinterrupting the sleep of the infant 602, the condition of the airquality in the sleep environment 604 can be linked as a possible causeof this behavior. Accordingly, the smart-home system can cause the airpurifier 608 to increase filtering operations to improve the air qualityin the sleep environment 604. The operation of the air purifier 608 canalso be influenced by the season and/or weather. During allergy season,the air purifier 608 can increase its operation to filter the air in thesleep environment 604 to remove pollutants and allergens, for example, asmall as 0.3 microns. The system can monitor the reaction of the infant602 in response to adjusting the operation of the air purifier 608 todetermine whether adjusting the operation of the air purifier 608improved the sleep conditions of the infant 602. Thus, the smart-homesystem can learn over time whether links between environmentalconditions and specific subject behaviors are correct, and whetherspecific adjustments made to environmental conditions improved thesubject behavior during sleep. This closed-loop feedback system can ineffect learn the types of environmental adjustments that are mosteffective for a particular subject over time.

The same child behaviors can sometimes be linked to differentenvironmental conditions. In the example above where coughing, sneezing,wheezing, labored breathing, etc., are detected, the smart-home systemcan attempt to adjust the air quality through the air purifier 608.Additionally or alternatively, these behaviors can also be linked toother environmental conditions, such as humidity. Therefore, detectingthese behaviors can also cause the smart-home system to adjust theoperation of a humidifier in the sleep environment 604 to increase thehumidity level in the sleep environment 604. As different environmentaladjustments are applied over time, the smart-home system can monitor thereactions in the behavior of the subject to determine which adjustmentsare most effective. For example, the smart-home system can, over time,determine whether a coughing infant 602 is most likely to be soothed byincreasing air purification, adjusting a humidity level, and/or both.

In another example, the ambient noise level in the sleep environment 604can be monitored. Particularly in a home with other children, noisesfrom outside the sleep environment 604 may enter the sleep environment604 and disturb the infant 602. The microphones on various smart-homedevices in the sleep environment 604 can monitor the sleep environment604 for loud noises that are permeating the sleep environment 604. Ifthese loud noises cause the sleep of the infant 602 to be interrupted,the smart-home environment can determine that a link exists between thenoise level and the behavior of the subject. In response, the smart-homeenvironment can increase and/or activate soothing noises in the sleepenvironment 604 to counteract or cover up the noises originating fromoutside the sleep environment 604. For example, the smart-homeenvironment can cause the home assistant 610 to play white noise,simulated rain, simulated running water, birds sounds, wind, lullabies,classical music, ocean sounds, and/or other noise that has beendetermined to help subjects go back to sleep. These noises can be playedby any smart-home devices in the sleep environment 604 that are equippedwith a speaker, such as the hazard detector 104. In some embodiments,these noises can be played by multiple smart-home devices simultaneouslyto provide a surround-sound effect in the room to better drown outexternal noise.

In some embodiments, smart-home devices outside of the sleep environment604 can detect sounds or detect events that will cause sounds to begenerated. This information can be communicated to the smart-homedevices in the sleep environment 604 to preemptively create, forexample, white noise to drown out these impending sounds from outsidethe sleep environment 604. For example, a smart home system can detect avisitor approaching the entrance of the home. Anticipating that thisevent will generate a doorbell sound, the home assistant 610 in thesleep environment 604 can generate white noise prior to the doorbellsounding to drown out the doorbell. In some embodiments, smart homesystems can provide crowd-sourced information to the server 164 that canbe used in nearby locations. For example, if a large truck is travelingdown the residential street, the sound made by the truck can be detectedby a doorbell system or security sensor in a home at one end of thestreet. This information can be transmitted through the server 164 toother homes on the street to generate white noise in the sleepenvironments in those homes to drown out the noise of the truck. Someembodiments may also use schedules of smart-home appliances toresponsively schedule counteracting noise generation in the sleepenvironment 604. For example, the smart-home system may determine thatan appliance just outside of the sleep environment 604 is about to turnon. Prior to turning on the appliance, the home assistant 610 cangenerate counteracting white noise in the sleep environment 604.

In some embodiments, the temperature of the infant 602 and/or thethermal signature of the infant can be used to control adjustments tothe setpoint temperature in the sleep environment 604. FIG. 13illustrates an image similar to that of FIG. 8 captured by the thermalimager function of the camera 118, according to some embodiments. Thethermal image represents a view of the thermal heat energyemitted/reflected by the infant 602. The thermal image has the advantageof not being dependent upon absolute lighting or variations in lightingfor image quality. Instead, the color bands in a facial image indicatedifferent heat bands in the skin of the infant 602. Each individualemits different patterns of thermal energy according to theirtemperature and facial characteristics. The typical temperature range ofthe human face/body is often quite uniform on the surface of the skin,varying from 35.5° C. to 37.5° C. The thermal patterns of the infant'sface 602 are derived primarily from a pattern of superficial bloodvessels that reside under the skin of the infant 602. The vein andtissue structure of every face is unique for each individual, andtherefore the thermal images generated of the face of each infant arealso unique.

When presented in a thermal image, the slight variations in temperatureon the skin of the infant 602 can be represented using different colorbands. Colder portions of the skin can be represented with darkercolors, such as colors closer to the blue/black end of the colorspectrum. Warmer portions of the skin can conversely be represented withlighter colors, such as colors closer to the white/yellow side of thecolor spectrum. The full-color spectrum in the thermal image can bescaled such that it covers the expected range of temperatures visible onthe skin of the infant 602. In FIG. 8, the darker/denser fill patternsof each color band are inversely proportional to the temperaturemeasured by the thermal imager. For example, the cheeks of the infant602 in FIG. 8 are colder than the area surrounding the eyes of theinfant 602.

In some embodiments, the thermal image of the infant 602 can be used toidentify the infant 602. As described above, a facial recognitiontechnique using the thermal image can be compared to known images in alocal memory and used to determine an identity for the infant 602. Someembodiments may allow users to register different infants or monitoredsubjects with the smart-home environment through a training processwhere the camera 118 automatically recognizes a subject that has notbeen seen before and alerts the user. The user can then provide anidentifier (e.g., a name) for the new subject. This can be particularlyadvantageous in homes or environments with multiple children or subjectsthat will be monitored. Identities of subjects can be stored locally andsecurely at the smart-home system.

In some embodiments, a baseline thermal signature can be determined forthe infant 602. Establishing a baseline thermal signature can be done ina number of different ways. In some embodiments, a baseline thermalsignature can record an average thermal image of portions of theinfant's exposed skin 602. First, the algorithm can identify exposedskin of the infant 602 by finding the warmer areas of the image. Exposedskin will generally emit more heat energy than clothed areas of theinfant 602. The algorithm can then record an image of the exposed skinas a baseline image. During a learning interval, such as one week, twoweeks, one month, etc., the baseline image can be combined/compared withsubsequent images to generate an average thermal signature for theinfant 602. As used herein, the term “thermal signature” can refer toany thermal characteristic of the infant 602 that may be recorded as abaseline and compared to future thermal images or characteristics. Inthis example, the thermal signature may be an average thermal image ofthe face of the infant 602, or a metric derived from the average facialimage of the infant 602.

In some embodiments, a baseline thermal signature can include anestimated internal temperature of the infant 602. In some embodiments,the average thermal image of the infant 602 can be assumed to be withina normal, healthy range of internal infant temperatures. In someembodiments, an algorithm has been developed to estimate the internaltemperature of the infant 602 based on the thermal image. This algorithmcomprises a method that does not require contact and at a remotedistance can estimate the internal body temperature of the infant 602 orany other object in the field-of-view of the camera 118. Because thesurface temperature is not necessarily consistent with the internaltemperature, this algorithm requires a new form of temperature analysis.Specifically, this algorithm derives a transfer function based on adistance of the camera 118 to the infant 602. The algorithm thendetermines an ambient temperature around the infant 602. For example,using the facial identification routine described above, the algorithmcan identify an area around the perimeter of the infant 602, such aslocation 804. Next, the algorithm can use temperature values from thethermal imager to determine the hottest spot on the skin of the infant602, such as location 802. Location 802 is most likely closest to theinternal temperature of the infant 602. The transfer function can thenbe computed using this temperature differential to provide an estimateof the inter-ear temperature of the infant 602.

In some embodiments, the system can also filter out thermal signaturesthat are not of interest or associated with the monitored subject. Forexample, some embodiments may filter out other heat signatures in theroom, such as space heaters, areas surrounding heat vents, while areasthat include hot-water pipes, heat-generating electronics, and otherheat sources that are not associated with the infant 602. Although theseother heat sources may be visible in the live video feed itself, theycan be filtered from the thermal image such that they do not interferewith the comparison algorithm described below for determiningheat-related characteristics of the infant 602.

In some embodiments, the resolution of the captured video feed can bealtered based on the bounding box 902 in FIG. 9. For example, the cameracan record and transmit a lower resolution video image for portions ofthe image outside of the bounding box 902, while preserving ahigh-resolution digital image for portions of the video feed that areinside the bounding box 902. This can simplify the video processing of athermal-signature-matching algorithm, decrease the bandwidth required totransmit the live video feed in real time, reduce the amount ofprocessing power required to analyze temperature-related anomalies inthe live video feed, and/or reduce the amount of memory required tostore images and information associated with the live/thermal videofeeds.

FIG. 14 illustrates a thermal image that can diagnose an infant 602 thatis to cold, according to some embodiments. In this example, certainareas of the image may be colder than the recorded baseline thermalsignature. For example, the nose 1402 of the infant 602 may appear to bea darker color in the thermal image than in the baseline image.Similarly, an exposed arm 1404 of the infant 602 may appear to be adarker color than in the baseline image. When certain areas of theinfant 602 are colder than the baseline image, this can indicate thatthe infant 602 is not warm enough. As described below, this situationcan generate an alert for a user indicating that the infant 602 may becold. Additionally or alternatively, this can generate automatic controlsignals for the smart-home environment to change the environmentalcharacteristics of the area surrounding the infant 602. For example,when the camera 118 detects areas of the infant 602 that are colder thannormal, the smart-home environment can generate commands for athermostat to increase the setpoint temperature of the room of theinfant 602. Thus, the colder surface temperature of the skin of theinfant 602 can be considered an infant behavior that can be linked tothe temperature condition of the sleep environment 604. The thermostat102 can responsively adjust the temperature in the sleep environment 604to warm the infant 602. It should be noted that this operation can bedone before the infant sleep is actually interrupted.

In some embodiments, a thermal image can be used to diagnose an infant602 who is teething, suffering from an infection, suffering from afever, and/or other medical conditions. When the infant 602 is teething,excessive heat may be generated around the mouth of the infant 602. Thisheat can be visible to the thermal imager around the mouth of the infant602, appearing lighter in color than normal. As was the case above, thecurrent thermal image of the infant 602 can be compared to a baselineimage from a baseline thermal signature, and differences can be isolatedand used to diagnose certain conditions. Generally, when a particularisolated area of the infant 602 is warmer than normal, this can indicatedifferent medical conditions, such as infections, ear infections,teething, etc. The algorithm can compare the thermal image of a baselinethermal signature with a current thermal image and identify areas thatare warmer by a predetermined threshold amount. As described below, thiscan generate an alert or informational indication that can be sent tothe mobile device of a parent or other user.

FIG. 15 illustrates a thermal image that can be used to diagnose aninfant 602 who is too warm, according to some embodiments. Generally,when an infant is too warm, the thermal pattern on the skin will belighter in the thermal image as depicted in FIG. 15. The behavioralcondition of being too warm can be linked to environmental condition ofthe setpoint temperature in the sleep environment 604 being set toohigh. In response, the smart-home system can cause the thermostat 102 tolower its setpoint temperature to help reduce the skin temperature ofthe infant 602. This adaptive temperature system can accommodate thedifferent sleep styles that may be used by parents. For example, someparents tend to over-swaddle the infant 602 or place too many clothes onthe infant 602 (e.g., beanie, sleep sack, onesie, etc.). The smart-homesystem can accommodate these different sleep conditions by adjusting thetemperature accordingly. Over-swaddled infants can be made comfortableby lowering the temperature in the sleep environment 604. Alternatively,some parents may minimize the amount of clothing and/or blankets in thecrib 606 out of caution. The smart-home system can accommodate thesesleep conditions by increasing the temperature in the sleep environment604 automatically. Some embodiments may use computer vision algorithmsto identify blankets, hats, and/or other clothing items placed on theinfant 602 to help diagnose that the temperature fluctuation on the skinof the infant 602 is due at least in part to such blankets, hats,clothing, etc. In some embodiments, the smart-home system can send analert to the parents on a mobile device indicating how the temperaturehas been adjusted, and how such adjustments can be avoided by changingthe clothing/bedding style of the infant 602 during sleep time.

In some embodiments, an elevated temperature in a relatively cool sleepenvironment for infant 602 that is not over-swaddled can indicate afever condition. When the infant 602 has a fever, the exposed skin inthe thermal image, particularly the skin of the face of the infant 602,will be lighter in color, indicating that the infant is warmer thannormal. A fever condition can be diagnosed by comparing the currentimage to a thermal image of the baseline thermal signature for theinfant 602. When a temperature differential is detected above apredetermined threshold amount (e.g., 2°, 3°, etc.) a fever diagnosiscan be communicated to a parent or other user. In addition tocalculating a temperature differential by comparing thermal images, aninternal temperature can be estimated for the infant 602. The methoddescribed above using a transfer function that incorporates the distancefrom the camera to the infant 602, the ambient room temperature, and awarmest estimated skin temperature can be used to estimate a currentinternal temperature of the infant 602. This estimated temperature canbe compared in some cases to the baseline temperature to ensure that thediagnosis is correct.

Turning back to FIG. 12, the method may include executing thecorresponding environmental adjustments (1210). As described above, thismay include changing setpoint temperatures, changing parameters thatgovern air filter operation, changing lighting conditions, changingsound generation patterns, and so forth. The change in the behavior ofthe subject in response to these changed environmental controls can bemonitored, and determinations can be made as to whether or not they wereeffective responses to the observed behaviors. Even though defaultvalues may be used initially as downloaded from the server 164 or storedon the smart-home devices, these default values can be changed over timeto be customized to the sleep habits of the particular infant 602 in theparticular sleep environment 604.

In some situations, the observed behavior of the infant 602 might not beassociated with any particular environmental condition or control. Forexample, a baby may become fussy during a sleep cycle due to bad dreams,digestive problems, growing pains, stomach discomfort, needing a diaperchange, and/or other problems that are not directly linked to anenvironmental condition. In these situations, the system can activateadditional sleep aids to help the infant 602 go back to sleep (1212). Asdescribed above, the sleep aids can be employed at the beginning of thesleep mode to help the infant 602 initially go to sleep. If the sleep ofthe infant 602 is interrupted or about to be interrupted during thesleep mode, the sleep aids can again be activated for a time interval.Note that in some embodiments, sleep aids can be activated in additionto making adjustments to control the environmental systems. For example,if the infant 602 is too warm, the smart-home system can adjust thesetpoint temperature of the thermostat 102 and activate the sleep aidsto help soothe and cool down the infant 602 at the same time.

FIG. 16 illustrates how some of the sleep aids can be activated in thesleep environment 604, according to some embodiments. One such sleep aidmay include generating sleep-inducing noise as described above. Thehazard detector 104, the camera 118, the home assistant 610, and/or anyof the other smart-home devices equipped with a speaker can generatethis sleep-inducing noise. The human brain is still able to perceive andprocessed sound in the auditory cortex while asleep. Sounds that arerelaxing, repetitive, and/or low-level can stimulate the auditory cortexin a way that induces sleep. Such sleep noises may include white noisethat approximates a combination of all noise frequencies and can be usedto mask other sounds. Other sounds may include simulated appliancenoises, such as compressors, refrigerators, vacuum cleaners, washingmachines, dryers, dishwashers, and so forth. Other sounds may includenature sounds, such as ocean waves, rain forest animals, thunderstorms,rain, wind, bird sounds, and so forth. Other sounds may include music,such as classical music, lullabies, vibraphone or mellowphone music, orother songs with which the infant 602 might be familiar.

When turning on and/or off, the smart home devices generating thesleep-inducing noise can do so gradually. This can prevent abruptlyadding new sounds to the sleep environment 604 or abruptly removingsoothing sounds from the sleep environment 604. In some embodiments,multiple smart-home devices can generate different parts of thesleep-inducing sound. For example, bird sounds may be generated by thehazard detector 104, the camera 108, and the home assistant 610. Each ofthese smart-home devices can generate individual bird noises, giving theeffect of an immersive outdoor environment in nature. This effect canalso surround the infant 602 with sound in a way that is bettercalculated to drown out external noises from outside the sleepenvironment 604.

In some embodiments, the activation of sleep aids may send an alert to,for example, the smart phone of a parent. The parent may then be able toselect specific noises to be played in the sleep environment 604. Theselections can influence the future automatic operation of thesmart-home system. Specifically, the smart-home system can learn fromthe selections made by the parent to automatically play specified soundsduring different portions of the sleep cycle of the infant 602. Forexample, the smart-home system can learn that the parent prefers alullaby to be played during the first 15 minutes of the sleep cycle, andprefers white noise to be generated during the remainder of the sleepcycle if sleep is interrupted.

Some embodiments may allow the smart-home devices to act as an intercomsystem between the sleep environment 604 and the mobile device of theparent or a home intercom system of the smart home. For example, whenthe infant 602 begins to wake up, an alert can be sent to the computingdevice of the parent giving the parent the opportunity to not onlyselect noises to be played in the sleep environment 604, but to alsotalk to the infant 602. Through the computing device, the parent cantalk to the infant 602, sing a lullaby to the infant 602, and/or provideany other comforting song or dialogue to help the infant 602 go back tosleep or sleep more soundly.

FIG. 17 illustrates how light can be used as a sleep aid to auto-soothethe infant, according to some embodiments. In addition to providingsound, the hazard detector 104, the camera 118, and/or the homeassistant 610 can provide lighting effects that can be used to helpsoothe the infant 602. In some embodiments, the thermostat 102 may alsobe equipped with an electronic display that can generate light and/orimages in conjunction with the other smart-home devices. In addition tothe smart-home devices depicted in FIG. 17, the sleep environment 604may also include other lighting devices that are controlled by thesmart-home system, such as smart lightbulbs or smart outlets thatcontrol other lighting appliances.

In some embodiments, when the infant 602 begins to stir in the crib 606,one or more of the smart-home devices equipped with lights can emit alow-level light to provide a nightlight. This can help soothe the infant602 by allowing them to see just enough of their surroundings to becomecomfortable in the sleep environment 604. In some embodiments,animations or light patterns can be displayed by the one or moresmart-home devices. For example, the display of the thermostat 102 canplay animations that may be visible to the infant 602. The lights of anyof the smart-home devices can activate in a coordinated fashion toactivate in a sequenced pattern to provide an animation effect to theinfant 602. In some embodiments, the lighting can be coordinated withthe sound emitted from the smart-home devices. For example, lights canbe activated when devices play their music in a surround-soundsimulation. In another example, lights can be activated in sequence withthe rhythm or beats of music being played by the smart-home devices.

In some embodiments, the smart-home devices can activate in situationsother than the “auto-soothe” situation when acting as a sleep aid. Forexample, some lights on the smart-home devices can be active all thetime to provide a nursery nightlight for the sleep environment 604. Inanother example, some lights can automatically and/or gradually turn onin a low-light mode when a parent wants to check on the infant 602. Thesmart-home system can detect when a parent opens the door through thecamera 118, a security system device, and/or by monitoring a location ofa smart-home device, such as a smart phone, that is carried by theparent. The parent may also provide inputs that turn on the nurserylighting so that they can check on the infant 602. Generally, thenursery lighting setting will be low enough that the sleep of the infant602 is not disturbed, while also providing enough light for the parentto navigate the sleep environment 604 and see the infant 602 whilesleeping. Additionally, motion sensors, such as the motion sensor in thehazard detector 104, can detect a parent moving across the sleepenvironment 604 and activate the nursery lighting setting.

This can be done while the system is in the sleep mode to avoid wakingup the infant 602. The system can then determine when the parent hasleft the room and allow the light to persist for short time. Afterwards,the light in the room can gradually fade out to avoid disturbing theinfant 602.

Although not depicted explicitly in FIG. 17, the sleep environment 604may include additional smart-home devices that can provide a soothingfunction for the infant 602 while sleeping. In some embodiments, thesleep environment 604 may also include a motorized mobile or otherinfant toy that can be mounted to the crib 606 and activated to soothethe infant 602 when their sleep is disturbed. Such mobiles may includetoys, designs, shapes, and/or the like, that rotate above the crib 606.In some embodiments, the sleep environment 604 may also include avibration device that causes the mattress or crib 606 to vibrate gentlyto help soothe the infant 602 to sleep. Different sleep environments mayinclude sleep areas other than the crib 606, such as chairs, electricswings, sleep pads, and so forth. Many of these sleep areas may includea vibrating or motion function that can be turned on by the smart-homesystem as part of a sleep aid or auto-soothing routine.

Turning back briefly to FIG. 12, if the sleep aids (1212) and/or theenvironmental adjustments (1210) are successful in helping the infant602 go back to sleep (1214), then the method may continue monitoring thebehavior of the infant 602 (1204) until the infant 602 wakes up and/orthe sleep of the infant 602 is disturbed again. In some cases, despitethe use of the sleep aids and/or environmental adjustments, the infant602 may not go back to sleep on their own. In these cases, thesmart-home system can send an alert to a computing device of asupervisor, parent, guardian, etc., informing them of the actions takenby the smart-home system and/or informing them that the infant 602 hasnot gone back to sleep on their own (1216).

FIG. 18 illustrates a system diagram for processing and transmittingimages and information between the smart-home devices 1810 in the sleepenvironment 604 and a user's mobile device 166, according to someembodiments. In this simplified diagram, some well-understood electroniccomponents may be omitted, such as a power outlet, an ethernet cable, aWi-Fi home router, and so forth. In some embodiments, the smart-homedevices 1810 can capture information regarding the infant 602 and/or thesleep environment 604. For example, the camera 118 may capture a livevideo feed 1802 of a monitored subject and perform processing operationsat the camera 118 itself. As described above, the camera 118 may includeone or more processors and one or more memory devices that can be usedfor executing predefined algorithms on the individual frames of thelive/thermal video feed. In the example described above, this mayinclude analyzing a thermal image of the infant 602 to diagnoseenvironmental problems or medical conditions. The camera 118 can sendresults of this diagnosis to the server 164. Additionally oralternatively, the camera 118 can also send the thermal image data alongwith the live video feed 1802 to the server 164.

Other smart-home devices 1810 can also send sleep disturbance data 1804to the server 164. This sleep disturbance data 1804 may include any formof sensor reading, such as sound recordings, threshold violations,motion detection information, thermal signatures, facial expressioninformation, and so forth. In some embodiments, the smart home devices1810 may include processors like the camera 118, and may be configuredto perform data analysis on the sleep disturbance data 1804 beforeproviding it to the server 164. In other embodiments, the smart homedevices 1810 can send the sleep disturbance data 1804 in its raw form tothe server 164 for analysis.

In some embodiments, the live video feed 1802 and/or the sleepdisturbance data 1804 may be captured and transmitted from the smarthome devices 1810. A remote server 164 that is accessible over theInternet through a home Wi-Fi router can also perform the imageprocessing algorithms on the live video feed 1802 and/or the thermalimage data along with aggregating and analyzing the sleep disturbancedata 1804 from the other smart home devices 1810. In these embodiments,the camera 118 can be a high-resolution camera that does not necessarilyneed to include processors and memories sufficient to execute the motiondetection algorithms described above. The server 164 may include asmart-home device monitoring server that collects monitoring informationfrom smart-home devices in the smart-home environment. The server 164may also provide data synchronization and/or software upgrades to eachof the smart-home devices, including the camera 118, in the smart-homeenvironment. The server 164 can be owned and/or operated by amanufacturer of the smart-home devices, including the camera 118. Theserver 164 may include a dedicated user account for each smart-homeenvironment (e.g., each home). The server 164 may be referred to hereinas a smart-home device monitoring server. The server 164 may also be incommunication with computer systems of other entities, such as a utilityprovider computer system (e.g., an energy utility), a law-enforcementcomputer system, an emergency-response computer system, and so forth.The server 164 may also include memory locations assigned to eachparticular user account where a historical record of the live video feed1002 may be stored and/or archived for later retrieval by the user ofthe account.

The server 164 can transmit the live video feed 1802, the thermal imagedata, the sleep disturbance data 1804, and/or data analysis, along withany alerts, indications, and/or diagnoses calculated at the smart-homedevices 1810 and/or the server 164 to a mobile device 166 of the userassociated with the account on the server 164. The mobile device 166 mayinclude a smart watch 166-1, a smartphone 166-2, a laptop computer, atablet computer, a desktop computer, a personal digital assistant (PDA),an on-board car computer system, a digital home assistant (e.g., GoogleHome®), and/or any other computing device. In some embodiments, the livevideo feed 1802, the thermal image data, and/or the sleep disturbancedata 1804 can be transmitted directly from the camera 118 to the mobiledevice 166 without passing through the server 164, but rather through alocal wireless network, such as Bluetooth® network or a proprietarysmart-home network (e.g., Thread®). Some embodiments may also transmitonly the live video feed 1802 and/or the raw sleep disturbance data 1804to the mobile device 166 and allow the mobile device 166 to process thisinformation to diagnose environmental conditions, sleep disturbances,and/or medical conditions for the infant 602. Therefore, the operationsdescribed herein for analyzing the various data and generatingindications, alerts, and/or diagnoses can be performed at any of thesmart home devices 1810, the server 164, the mobile device 166, and/orany other processing system that is part of the smart-home environment.

FIG. 19 illustrates a representation of the thermal video feed 1904 asit is displayed on a mobile device 166-2. Seeing the actual thermalvideo feed 1904 can be useful for providing additional information aboutthe environmental condition or or condition of the infant 602 asdetermined by the smart-home system. For example, by seeing the thermalvideo feed 1904, a parent can immediately determine that the infant isswaddled too tightly, has too many blankets, or is not sufficientlycovered. This can help the parent or user distinguish between situationswhere the temperature in the sleep environment 604 is too high insituations where the infant 602 is covered with too many clothes,blankets, etc.

The thermal video feed 1904 can also be used in conjunction with anylinks to additional information in the alert 1902. For example, someembodiments can transmit thermal images of monitored subjects to theserver 164. When a particular condition is detected, the user can selecta link provided in the alert 1402 to see other images stored at theserver 164 such that they can visually compare these images to thecurrent thermal image of their own infant. For example, by seeingbaseline thermal images of their infant 602, the parent can have greaterconfidence in the abnormal environmental conditions or behavior detectedby the smart-home system that are based on the thermal image that theysee on their own mobile device 166-2. In another example, if the camera118 or smart home device 166-2 indicates that the infant has a medicalcondition such as a fever, seeing additional thermal images may help theparent identify environmental differences that may account for theraised temperature of the infant rather than a fever.

In some embodiments, the smart-home system can store a library ofhistorical thermal images of the infant that can be retrieved andcompared over time. This can provide a library of thermal images andestimated temperatures to the parent. Thus, the parent can see how thesleep condition of the infant has progressed day-to-day andweek-to-week. By comparing a history of images in the last hour, thealert 1902 can indicate to a user that the infant's temperature isbeginning to decrease. Similarly, a history of images can reveal that aninfection or teething situation is beginning to subside rather thanincrease. The alert 1902 can incorporate these findings and indicatethat the infant's condition appears to be improving. For example, analert can be provided not only when disturbed sleep is detected, butalso when the child's sleep returns to normal, indicating that thetemperature of the infant 602 is returning to normal, the air quality isimproving, and so forth.

The alert 1902 can also include links to additional information or to acontrol panel to further control the smart-home system. For example, thealert 1902 may include a link to a website that describes optimal sleepconditions for the infant 602. In another example, the alert 1902 mayinclude a link to medical information to help diagnose and treat asuspected medical condition. In another example, the alert 1902 mayinclude a link to a control panel in an app/application provided by thesmart-home system to control the smart-home devices in the sleepenvironment 604. The control panel may include different options thatthe parent can activate, such as any of the auto-soothing systemsdescribed above, including lighting, sound, moving devices, vibration,and so forth. The alert 1902 may also include a list of controloperations that have already been executed by the smart-home system inthe sleep environment 604. For example, the alert 1902 describes thecondition of the infant 602 (“your baby appears to be too warm”). Thealert 1902 also includes a list of actions that were taken by thesmart-home system (“your thermostat has been turned down from 76° to73°”). In some embodiments, the alert 1902 may include additionalactions or control operations that may be taken by selecting thoseoptions in the alert 1902. For example, be alert 1902 may include anoption to turn on additional sounds in the sleep environment 604 (“wouldyou like to activate the white noise generator?”). By selecting thisoption, the parent can activate additional smart-home systems in thesleep environment 604 as described above. Thus, not only does thesmart-home system automatically respond to the disrupted sleep of theinfant 602, but parents or guardians are given full control over thesystem in real-time to tailor the response of the smart-home system asthey see fit. When the parents or guardians provide these inputs, thesmart-some system can tailor its responses in the future to correspondto the actions taken in the past by the parents or guardians.

FIG. 20 illustrates a representation of the live video feed 2006displayed on a mobile device 166-2, according to some embodiments. Thelive video feed 1406 can be displayed on the screen of the mobile device166-2 as the user monitors the infant 602. In some cases, the infant 602may be monitored on the mobile device 166-2 for an extended period oftime, where real-time video is displayed to the user. This can be doneto enhance the emotional connection between the user and the infantthrough the mobile device 166-2. For example, a parent who needs to beaway from the infant during the day can log into the server 164 usingthe mobile device 166-2 and watch real-time video of the infant.

In some cases, the user can monitor the infant to watch for sleepdisturbances, and medical emergency conditions, and/or times when theinfant 602 wakes up. While normal video streams of previous systems inthe art would make it difficult or impossible to visually see or detectthese medical conditions, the camera system and thermal imager describedabove can make these medical conditions readily apparent to an observerof the mobile device 166-2. In addition to simply displaying thereal-time video feed 2006, the mobile device 166-2 can also displayvisual/audio warnings or status messages for any of the medicalconditions detected above. In the example of FIG. 20, an alert 2002 isdisplayed on the screen indicating that the sleep of the infant 602appears to be disturbed. The alert may include a vibration, a sound,and/or color encodings that indicate the severity of the alert. Forexample, a temperature elevated by more than 2° can generate an alert1402 that is colored red and generates an audible alarm and/or vibrationof the mobile device 166-2. Some embodiments may also generateindications that include additional information based on the behaviorthat is observed. For example, if the smart-home system indicates thatthe infant 602 is coughing, the alert 2002 can include a number of timesthat the infant has coughed, time intervals during which coughs havebeen detected, intensity of coughs, and so forth. As described above,the alert 2002 may include information regarding control actions thathave already been executed or which are planned to be executed by thesmart-home system. For example, the alert 2002 may indicate that thesmart-home system has turned on one or more nightlights and will beginplaying a white noise track. As described above, the alert 2002 may alsoinclude an option for the parent to personally check on the infant 602.By selecting this option, the smart-home system can automatically turnon one or more low-level lights in the sleep environment 604 to allowthe parent to see the infant 602 when they enter the sleep environment604.

FIG. 21 illustrates an alternative visual representation of an alert ona mobile device 166-1, according to some embodiments. In thisembodiment, the mobile device 166-1 may include a smart watch or otherpiece of wearable technology. This alert 2102 is similar to the alertillustrated in FIG. 20 indicating that the infant appears to bestirring. The display may also include a thermal image of the monitoredsubject. Alternatively or additionally, the mobile device 166-1 mayinclude the live video feed 2104 as depicted in FIG. 21. In anyembodiment, mobile devices 166 may allow the user to toggle back andforth between the images from the thermal video feed and the images fromthe live video feed.

The alert 2102 may include the option to allow the user to talk to theinfant 602 through the mobile device 166-1 and have their voicebroadcast by a one or more of the smart home devices in the sleepenvironment 604. A microphone on the mobile device 166-1 can allow theparent to, for example, sing to the infant 602, play a song for theinfant 602, tell a story to the infant 602, and so forth. Thus, thesmart-home system can establish a video/audio link between the sleepenvironment 604 and the mobile device 166-1. In some embodiments, thesleep environment 604 may include a monitor that is visible from thecrib 606 that displays video of the parent as captured by the mobiledevice 166-1, and may thereby provide a two-way video/audio feed for theinfant 602 and the parent to communicate.

FIG. 22 illustrates a closed-loop local feedback system for updatingdefault sleep-mode parameters, according to some embodiments. Asdescribed briefly above, the smart-home system can constantly belearning behaviors of the infant 602 and/or the appropriate responses totake in different situations such that the sleep environment 604 can beoptimized. The default parameters 2202 can be received from the serverand/or stored on the smart home devices 2204 as described above. Thesedefault parameters may represent parameters that have been found to beoptimal for controlling smart home devices for a population of infantsthat are similar to a particular infant 602. These default parameters2202 can be used to control the operation of the smart home devices 2204during the sleep mode.

As the system operates, it can monitor the effects of certain actionsthat are taken and the corresponding responses of the infant 602. Forexample, the default parameter for a temperature setpoint may be 75°.However, keeping the sleep environment 604 at 75° may cause the infant602 to become too warm according to the thermal images captured by thethermal imager of the camera 118. By linking this elevated temperatureto the control of the thermostat, the smart-home system can decrease thesetpoint temperature of the thermostat 102 down to 73°. The camera 118can then monitor that reaction of the infant 602 to the temperaturechange. The infant reaction can be represented by a Boolean value asbeing successful are not successful. Alternatively or additionally, theinfant reaction can use a more complex representation, such as a scalebetween successful and unsuccessful. Both the reaction of the infant 602and the local update to the parameter 2206 can be fed into a neuralnetwork or machine learning algorithm of a smart home controller device2208. Any of the smart home devices with a processor can act as thesmart-home control device 2208. Additionally, a data hub, a workstation,a laptop, and/or any other computing device can act as the smart-homecontrol device.

The neural network or machine learning algorithm of the smart-homecontrol device 2208 can monitor local parameter updates andcorresponding infant responses over time and generate changes to thedefault parameters. For example, if the thermostat is repeatedly turneddown from the default parameter value of 75°, and more often than notthe infant responds positively to this temperature change by going backto sleep, the neural network or machine learning algorithm can determinethat the default parameter representing the temperature setpoint of thethermostat 102 can be lowered incrementally. In the example of FIG. 22,the system can close the feedback loop by providing a default parameterupdate 2210 to the default parameters 2202. The updated defaultparameters 2202 can be used in the smart home devices 2204 the next timethey enter the sleep mode.

The closed-loop feedback system of FIG. 22 can use control inputs thatare automatically generated by the smart-home system. Additionally oralternatively, the feedback system can use the user inputs from parentsreceived through mobile devices. In some embodiments, user inputs can beweighed more heavily than control inputs that are automaticallygenerated by the system. As the system learns, automatic control actionsthat do not result in a successful outcome for the sleep behavior of theinfant can be reduced such that they are less likely to take place inthe future.

FIG. 23 illustrates a crowd-sourced feedback system for updating defaultsleep parameters for populations of similar subjects, according to someembodiments. The default parameters 2310 may originally be distributedto a plurality of smart home environments 2304 which may correspond todifferent homes, residences, buildings, enclosures, and so forth.

These default parameters 2310 can operate in the smart home environments2304 using the closed loop feedback system described above in relationto FIG. 22. As described above, the default parameters 2310 cancorrespond to clustered populations of monitored subjects having similarcharacteristics.

As the closed loop feedback systems for each of the smart-homeenvironments 2304 update the default parameter values, the smart homeenvironments 2304 can send local parameter updates 2302 back to theserver 164. A neural network 2306 can receive all the local parameterupdates 2302 and determine when a sufficient number of similar updateshave been received to generate a default parameter update 2308 for thedefault parameters 2310. For example, if the default parameters 2310represent a population of approximately 500 subjects having similarcharacteristics, the neural network 2306 can generate the defaultparameter update 2308 when a threshold number (e.g., 50) of localparameter updates 2302 have been received for a particular parametervalue.

When the local parameter updates 2302 are received, the neural networkcan cause the default parameters 2310 to change in magnitude in thedirection of the average of the local parameter updates 2302.Alternatively or additionally, the neural network 2306 can generate anew cluster of similar individuals. For example, if the local parameterupdates 2302 all come from very similar subjects and/or sleepenvironments, the neural network can split the existing cluster into twoseparate clusters. The cluster associated with the local parameterupdates 2302 can use the same existing default parameters 2310 butincorporate the default parameter update 2308. The remainder of theprevious cluster can continue to use the default parameters 2310 withoutthe default parameter update 2308.

The local parameter updates 2302 may include characteristics of thesubject and/or the sleep environment when permission to share suchinformation is granted by the parents or other monitor of the subject.Information such as the subject name, address, etc. need not be sharedwith the server 164. Alternatively or in conjunction therewith, any of avariety of known data anonymization methods can be used to protect userprivacy, while at the same time providing statistically useful data forpurposes of achieving the features and advantages of the embodimentsdescribed herein.

It should be appreciated that the specific steps illustrated in FIG. 15provide particular methods of monitoring physical characteristics ofsubjects in sleep environments according to various embodiments of thepresent invention. Other sequences of steps may also be performedaccording to alternative embodiments. For example, alternativeembodiments of the present invention may perform the steps outlinedabove in a different order. Moreover, the individual steps illustratedin FIG. 15 may include multiple sub-steps that may be performed invarious sequences as appropriate to the individual step. Furthermore,additional steps may be added or removed depending on the particularapplications. One of ordinary skill in the art would recognize manyvariations, modifications, and alternatives.

In the foregoing description, for the purposes of explanation, numerousspecific details were set forth in order to provide a thoroughunderstanding of various embodiments of the present invention. It willbe apparent, however, to one skilled in the art that embodiments of thepresent invention may be practiced without some of these specificdetails. In other instances, well-known structures and devices are shownin block diagram form.

The foregoing description provides exemplary embodiments only, and isnot intended to limit the scope, applicability, or configuration of thedisclosure. Rather, the foregoing description of the exemplaryembodiments will provide those skilled in the art with an enablingdescription for implementing an exemplary embodiment. It should beunderstood that various changes may be made in the function andarrangement of elements without departing from the spirit and scope ofthe invention as set forth in the appended claims.

Specific details are given in the foregoing description to provide athorough understanding of the embodiments. However, it will beunderstood by one of ordinary skill in the art that the embodiments maybe practiced without these specific details. For example, circuits,systems, networks, processes, and other components may have been shownas components in block diagram form in order not to obscure theembodiments in unnecessary detail. In other instances, well-knowncircuits, processes, algorithms, structures, and techniques may havebeen shown without unnecessary detail in order to avoid obscuring theembodiments.

Also, it is noted that individual embodiments may have been described asa process which is depicted as a flowchart, a flow diagram, a data flowdiagram, a structure diagram, or a block diagram. Although a flowchartmay have described the operations as a sequential process, many of theoperations can be performed in parallel or concurrently. In addition,the order of the operations may be re-arranged. A process is terminatedwhen its operations are completed, but could have additional steps notincluded in a figure. A process may correspond to a method, a function,a procedure, a subroutine, a subprogram, etc. When a process correspondsto a function, its termination can correspond to a return of thefunction to the calling function or the main function.

The term “computer-readable medium” includes, but is not limited toportable or fixed storage devices, optical storage devices, wirelesschannels and various other mediums capable of storing, containing, orcarrying instruction(s) and/or data. A code segment ormachine-executable instructions may represent a procedure, a function, asubprogram, a program, a routine, a subroutine, a module, a softwarepackage, a class, or any combination of instructions, data structures,or program statements. A code segment may be coupled to another codesegment or a hardware circuit by passing and/or receiving information,data, arguments, parameters, or memory contents. Information, arguments,parameters, data, etc., may be passed, forwarded, or transmitted via anysuitable means including memory sharing, message passing, token passing,network transmission, etc.

Furthermore, embodiments may be implemented by hardware, software,firmware, middleware, microcode, hardware description languages, or anycombination thereof. When implemented in software, firmware, middlewareor microcode, the program code or code segments to perform the necessarytasks may be stored in a machine readable medium. A processor(s) mayperform the necessary tasks.

In the foregoing specification, aspects of the invention are describedwith reference to specific embodiments thereof, but those skilled in theart will recognize that the invention is not limited thereto. Variousfeatures and aspects of the above-described invention may be usedindividually or jointly. Further, embodiments can be utilized in anynumber of environments and applications beyond those described hereinwithout departing from the broader spirit and scope of thespecification. The specification and drawings are, accordingly, to beregarded as illustrative rather than restrictive.

Additionally, for the purposes of illustration, methods were describedin a particular order. It should be appreciated that in alternateembodiments, the methods may be performed in a different order than thatdescribed. It should also be appreciated that the methods describedabove may be performed by hardware components or may be embodied insequences of machine-executable instructions, which may be used to causea machine, such as a general-purpose or special-purpose processor orlogic circuits programmed with the instructions to perform the methods.These machine-executable instructions may be stored on one or moremachine readable mediums, such as CD-ROMs or other type of opticaldisks, floppy diskettes, ROMs, RAMs, EPROMs, EEPROMs, magnetic oroptical cards, flash memory, or other types of machine-readable mediumssuitable for storing electronic instructions. Alternatively, the methodsmay be performed by a combination of hardware and software.

1. A method of monitoring and optimizing sleep of a subject using aplurality of smart-home devices, the method comprising: operating asmart-home system comprising the plurality of smart-home devices,wherein the smart-home system is configured to operate in a plurality ofmodes comprising: a normal operating mode; and a sleep mode; determiningthat the smart-home system should transition into the sleep mode fromthe normal operating mode, wherein the plurality of smart-home devicesuse a set of default parameters when operating in the sleep mode;monitoring, while in the sleep mode, a sleep cycle of the subject usingthe plurality of smart-home devices; detecting, by the plurality ofsmart-home devices, behavior of the subject that indicates that thesleep cycle of the subject is being interrupted; determining anenvironmental control that corresponds with the behavior of the subject;adjusting the environmental control using the plurality of smart-homedevices, wherein the adjusting is configured to prevent or stop thesleep cycle of the subject from being interrupted; determining whetheradjusting the environmental control using the plurality of smart-homedevices prevented or stopped the sleep cycle of the subject from beinginterrupted; and adjusting the set of default parameters for use insubsequent instances where the sleep cycle of the subject is beinginterrupted, wherein the set of default parameters are adjusted based ona determination of whether adjusting the environmental control using theplurality of smart-home devices prevented or stopped the sleep cycle ofthe subject from being interrupted.
 2. The method of claim 1, whereinthe plurality of smart-home devices comprises a camera with a thermalimager.
 3. The method of claim 2, wherein detecting the behavior of thesubject that indicates that the sleep cycle of the subject is beinginterrupted comprises determining, using the thermal imager, that atemperature of the subject is above or below a target temperature. 4.The method of claim 3, wherein the thermal imager captures a thermalimage of a face of the subject and compares the thermal image of theface of the subject to a baseline thermal image of the face of thesubject.
 5. The method of claim 3, wherein: the plurality of smart-homedevices comprises a thermostat; and adjusting the environmental controlusing the plurality of smart-home devices comprises adjusting atemperature setpoint of the thermostat to bring the temperature of thesubject closer to the target temperature.
 6. The method of claim 1,wherein determining that the smart-home system should transition intothe sleep mode comprises determining, using a camera in the plurality ofsmart-home devices, that the subject is in a sleep environment that ismonitored by the plurality of smart-home devices.
 7. The method of claim6, further comprising determining that the subject is in the sleepenvironment in a location that indicates that the subject is intended tosleep.
 8. The method of claim 1, wherein determining that the smart-homesystem should transition into the sleep mode comprises receiving aninput from a user indicating that the sleep cycle of the subject isbeginning.
 9. The method of claim 1, wherein determining that thesmart-home system should transition into the sleep mode compriseslearning, by the smart-home system, a schedule of the sleep cycle of thesubject over time.
 10. The method of claim 1, wherein the set of defaultparameters comprises parameters that are associated with a population ofsubjects that are similar to the subject.
 11. A smart-home system formonitoring and optimizing sleep of a subject using a plurality ofsmart-home devices, the smart-home system comprising: a plurality ofsmart-home devices that are configured to operate in a plurality ofmodes comprising: a normal operating mode; and a sleep mode; one or moreprocessors; and one or more memory devices comprising instructions that,when executed by the one or more processors, cause the one or moreprocessors to perform operations comprising: determining that thesmart-home system should transition into the sleep mode from the normaloperating mode, wherein the plurality of smart-home devices use a set ofdefault parameters when operating in the sleep mode; monitoring, whilein the sleep mode, a sleep cycle of the subject using the plurality ofsmart-home devices; detecting behavior of the subject that indicatesthat the sleep cycle of the subject is being interrupted; determining anenvironmental control that corresponds with the behavior of the subject;adjusting the environmental control using the plurality of smart-homedevices, wherein the adjusting is configured to prevent or stop thesleep cycle of the subject from being interrupted; determining whetheradjusting the environmental control using the plurality of smart-homedevices prevented or stopped the sleep cycle of the subject from beinginterrupted; and adjusting the set of default parameters for use insubsequent instances where the sleep cycle of the subject is beinginterrupted, wherein the set of default parameters are adjusted based ona determination of whether adjusting the environmental control using theplurality of smart-home devices prevented or stopped the sleep cycle ofthe subject from being interrupted.
 12. The smart-home system of claim11, wherein the set of default parameters are downloaded from amonitoring server that communicates and processes data from a pluralityof smart-home systems.
 13. The smart-home system of claim 11, whereinthe set of default parameters cause the plurality of smart-home devicesto be configured to minimize external noise in a sleep environment forthe subject.
 14. The smart-home system of claim 11, wherein: thebehavior of the subject that indicates that the sleep cycle of thesubject is being interrupted comprises coughing or sneezing by thesubject; and adjusting the environmental control using the plurality ofsmart-home devices comprises adjusting an operation of an air purifieror humidifier.
 15. The smart-home system of claim 11, wherein theplurality of smart-home devices comprises a thermostat, a camera, and ahazard detector.
 16. The smart-home system of claim 11, wherein thesmart-home system further comprises a monitoring server, wherein the oneor more processors are distributed between the plurality of smart-homedevices and the monitoring server.
 17. The smart-home system of claim11, wherein the operations further comprise: determining that a user isentering a sleep environment of the subject; causing one or more of theplurality of smart-home devices to generate a low-level night light. 18.The smart-home system of claim 11, wherein the operations furthercomprise, after transitioning into the sleep mode, causing one or moresleep aids to be activated until it is determined that the subject isasleep.
 19. The smart-home system of claim 18, wherein the one or moresleep aids comprise one or more of the following: white noisegeneration, low-level lighting, music, vibrations, and mobileactivation.
 20. The smart-home system of claim 11, wherein theoperations further comprise adjusting the set of default parametersbased on adjusting the environmental control.